Target Propagation via Regularized Inversion
Authors: Vincent Roulet
Zaid Harchaoui
Venue: ICLR 2022 Submitted (2022)
Keywords: Target propagation
differentiable programming
recurrent neural networks

Abstract: Target Propagation (TP) algorithms compute targets instead of gradients along neural networks and propagate them backward in a way that is similar yet different than gradient back-propagation (BP). The idea was first presented as a perturbative alternative to back-propagation that may achieve greater accuracy in gradient evaluation when training multi-layer neural networks (LeCun et al., 1989). However, TP has remained more of a template algorithm with many variations than a well-identified algorithm. Revisiting insights of LeCun et al., (1989) and more recently of Lee et al. (2015), we present a simple version of target propagation based on regularized inversion of network layers, easily implementable in a differentiable programming framework. We compare its computational complexity to the one of BP and delineate the regimes in which TP can be attractive compared to BP. We show how our TP can be used to train recurrent neural networks with long sequences on various sequence modeling problems. The experimental results underscore the importance of regularization in TP in practice.

Under review as a conference paper at ICLR 2022
TARGET PROPAGATION VIA REGULARIZED INVERSION
Anonymous authors
Paper under double-blind review
ABSTRACT
Target Propagation (TP) algorithms compute targets instead of gradients along neural networks and propagate them backward in way that is similar yet differ- ent than gradient back paok-propagation (BP). The idea was first presented as a pertur- bative alternative to back- propagatin that may improve gradient evaluation accu- racy when training multi-layer neural networks (Le Cun et al., 1989). However, TP may have remained more of template algorithm with many variations than a well-identiffed algorithm. Revisiting insights of Le Cun et al. (1989) and more recently of Lee al. (2015), we present simple version of target propagation based on a regularized inversion of network layers, easily implementable in dif- ferentiable programming framework. We compare its computational complexity to the one of BP and delineate the regimes in which TP can be attractive compared to BP. We show how our TP can be used to train recurrent neural networks with long sequences on various sequence modeling problems. The experimental results underscore the importance of regularization in TP in practice.
INTRODUCTION
Target propagation algorithms can be seen as perturbative learning alternatives to the gradient back- propagation algorithm, where virtual targets are propagated backward instead of gradients (Le Cun, 1986; Le Cun et al., 1989; Rohwer, 1990; Mirowski & LeCun, 2009; Bengio, 2014; Goodfellow al., 2016). A high-level summary is presented in Fig. 1: while gradient back-propagation con- siders storing intermediate gradients in forward pass, target propagation algorithms proceed by computing and storing approximate inverses. The approximate inverses are then passed on back- ward along the graph of computations to finally yield weight update for stochastic learning.
Target propagation aims to take advantage of the availability of approximate inverses to compute better descent directions for the objective at hand. Bengio et al. (2013); Bengio (2020) argued that the approach could be relevant for problems involving multiple compositions such as the training of Recurrent Neural Networks (RNNs), which generally suffer from the phenomenon of exploding or vanishing gradients (Hochreiter, 1998; Bengio et al., 1994; ?). Recently, empirical results indeed showed the potential advantages of target propagation over classical gradient back-propagation for training RNNs on several tasks (Manchev & Spratling, 2020). However, these recent investigations remain built multiple approximations, which hinder the analysis of the core idea of TP, i.e., using layer inverses.
On the theoretical side, difference target propagation, a modern variant of target propagation, was related to an approximate Gauss-Newton method, suggesting interesting venues to explain the ben- efits of target propagation (Bengio, 2020; Meulemans et al., 2020). Previous works have considered approximating inverses by adding multiple reverse layers (Manchev & Spratling, 2020; Meulemans al., 2020; Bengio, 2020). However, is unclear whether such reverse layers actually learn layer inverses during the training process. Even if they were, the additional cost of computational com- plexity of learning approximate inverses should be carefully accounted for.
In this work, we propose a simple target propagation approach, revisiting the original insights of Le Cun et al. (1989) on the critical importance of the good conditioning of layer inverses. We define regularized inverses through variational formulation and we obtain approximate inverses via these regularized inverses. In this spirit, we can also interpret the difference target propagation formula (Lee et al., 2015) finite difference approximation of linearized regularized inverse. We propose a smoother formula that can directly be integrated into a differentiable programming framework.

---

Under review as a conference paper at ICLR 2022
Gradient Back-Propes Pack-Propagttion
Target Propagation
Forward Pass
f
f
@f())
(v)
Backward Pass
0f(x)
0
(1)
Fig. 1: Our implementation of target propagation uses lincarization of gradient inverses instead of gradients in a backward pass akin to gradient back- propagation..
We detail the computational complexity of the proposed target propagation and compare it to the one of gradient puck-propagation, showing that the additional cost of computing inverses can be effectively amortized for very long sequences. Following the benchmark of Manchev & Spratling (2020), observe that the proposed target propagation can perform better than classical gradient- based methods on several tasks involving RNNs.
Related work. Many variations of back- propagation algorithms have been explored; see Werbos (1994); Goodfellow et al. (2016) for an extensive bibliography. Closer target propagation, pe- nalized formulations of the training problem have been considered to decouple the optimization of the weights in distributed way using an ADMM approach (Carreira-Perping Perrinnn & Wang, 2014; Taylor et al., 2016; Gotmare et al., 2018). Rather than modifying the backward operations in the layers, one can also modify the weight updates for deep forward networks by using a regularized inverse (Frerix et al., 2018). Wiseman et al. (2017) recast target propagation as an ADMM-liKe algo- rithm for language modeling and reported disappointing experimental results. Recently, in a careful experimental benchmark evaluation, Manchev & Spratling (2020) explored further target propaga- tion to train RNNs, mapping sequence to single final output, in an attempt to understand the ben- efits of target propagation to capture long-range dependencies, and obtained promising experimental results. Another line of research has considered synthetic gradients that approximate gradients us- ing an additional layer instead of using back propppggtttted gradients (Jaderberg et al., 2017; Czarnecki et al., 2017) to speed up the training of deep neural networks. Recently, Ahmad et al. (2020); Dalm al. (2021) considered using analytical inverses implement target propagation and blend it with what they called a gradient-adjusted incremental formula. Yet, an additional orthogonality penalty critical for their approach work. Recently, Meulemans al. (2020) considered using as many reverse layers as forwarding operations. We focus here on the optimization gains of using target propagation that cannot be obtained by adding a prohibitive number of reverse layers. Finally, we do not discuss the biological plausibility of TPP since we are unable to comment on this. We refer the interested reader to, e.g., (Bengio, 2020).
Notations.
For f RP x R -> Rd, we denote 0zf(x, y) (2))0))))) E Rdxx
2 TARGET PROPAGATION WITH LINEARIZED REGULARIZED INVERSES
While target propagation was initially developed for multi-layer neural networks, we focus on its implementation for recurrent neural networks, as we shall follow the benchmark of Manchev & Spratling (2020) in the experiments. Recurrent Neural Networks (RNNs) are also canonical family of neural networks in which interesting phenomena arise back propagattaaanonnn algorithms.
Problem setting. A simple RNN parameterized by 0 (Wah, Wzh, by. Why- by) maps se- quence of inputs æ1:r (Z1...., x,) to an output (o( by computing hidden states he E IR corresponding to the inputs Zt.
Formally, the output I and the hidden states h are computed as an output operation following transition operations defined as
" co (hr) = s(Wnyhe + by),
h4 (h4-1) = aWWaaa + Whihh-1 bw) fort E (1,...,7);
2

---

Under review as a conference paper at ICLR 2022
where is, e.g., the soft-max function for classification tasks, is non-linear operation such as the hyperbolic tangent function, and the initial hidden state is generally fixed as ho = Given samples of sequence-output pairs (Z):r y), the RNN is trained to minimize the error Ely, go (x1:r)) of predicting ù = (o(r:)) instead of y-
As one considers longer sequences, RNNs face the challenge of exploding/vanishing gradients (2:r)/)))/ (Bengio & Frasconi, 1995); see Appendix for more discussion. We acknowl- edge that specific parameterizzatinn-aase
strategies have been proposed address this issue of ex- bhoding/vanishinn gradients, such as orthonormal parameterizations of the weights (Arjovsky et al., 2016: Helfrich et al., 2018: Lezcano- Casaoo & Marttnez-Rubl 2019). The focus here is to simplify and understand target propagation as backpropagation-tyyn algorithm using RNNs as a workbench. Indeed, training RNNs is an optimization problem involving multiple compositions for which ap- proximate inverses can easily be available. The framework could also be potentially applied to, e.g., time-series or control models (Roulet al., 2019).
Given the parameters Wa,, Wan, by of the transition operations, we can get approximate inverses of (o.((t-1) ) for all E : that yield optimization surrogates that can be better perform- ing than the ones corresponding to regular gradients. We present below a simple version of target propagation based on regularized inverses and inverse linearizations.
Back-propagating targets. The idea of target propagation is to compute virtual targets e, for each layer = r,.... such that if the layers were able to match their corresponding target time f, i.e., (h x Us, the objective would decrease. The final target U, is computed as a gradient step on the loss w.r.t. ht. The targets are then back-propagated using an approximate inverse' fø.t of Sa, at each time step.
Formally, consider RNN that computed T states hi...., h, from sequence 21..... ................................ 2y with associated output y. For a given stepsize "/, > 0, we propose to vack-propagate targets by computing
Ur = h, mn (n,
(1)
Vt-1 = - 0nf (he) (v--h,) h.), for t E {T,...,
+
(2)
The update rule (2) blends two ideas: i) regularized inversion; ii) linear approximation. We shall describe below that our update (2) allows us to interpret the "magic formula" of difference target propagation in Eq. 15 of Lee et al. (2015) as Oth-order finite difference approximation, while ours 1st-order linear approximation. We shall also show that (2) puts in practice an insight from Bengio (2020) suggesting to use the inverse of the gradients the spirit of a Gauss-Newton method.
Once all targets are computed, the parameters of the transition operations are updated such that the outputs of fe,t at each time step move closer to the given target. Formally, the update consists of a gradient step with stepsize tt on the squared error between the targets and the current outputs, i.e.. for Og, E (Whr. hh: W. aho b,
6 = 0 70
So (ho.)() - - - 1
(3)
As for the parameters 0g- = (Ways ky: by) of the output operation, they are updated by a simple gradient step on the loss with a stepsize 7o.
2.1 REGULARIZED INVERSION
To explore further the original idea of Le Cun et al. (1989), we consider using the variational defini- tion of the inverse,
(vx) = argmin [l. - vellz = argmin |a + WAA + b,) vol. T.t
(4)
ERP
As long belongs to the image (RP) of fe.. this definition recovers exactly the inverse
of Ut by fe.t More generally, if v & fo. Eq. (4) computes the best approximation of the
In the following, to case the presentation, we abuse notations and denote approximate inverses by forl-
3

---

Under review as a conference paper at ICLR 2022
inverse in the sense of the Euclidean projection. When considers activation function and 8, = (Wins Wan, b).), the solution of (4) can easily be computed.
Formally, for the sigmoid, the hyperbolic tangent the ReLU, their inverse can be obtained analyt- ically for any Ds E a(R²). So for v4 E ((RP) and Whh full rank, we get
So (ve) (Wi. kh Wan) - W (a (vt) - bn).
If th E a (RP)) the minimizer of (4) is obtained by first projecting te onto (RP). before inverting the linear operation. To account for non-invertible matrices Was. we also add regularization in the computation of the inverse. Overall we consider approximating the inverse of the layer by regularized inverse of the form
fø. (v)) (W Whs +r1) +
WT Ah (a (x(()))()))))) Wa bk).
with > 0 and a projection onto a(RP).
Regularized inversion vs. parameterized inversion. Bengio (2014); Manchev & Spratling (2020) parameterize the inverse as reverse layer such that
fo,t (we) = (vx) == a (Wank Vu, + cc,
and learn the parameters 0' (V, c) for this reverse layer approximate the inverse of the forward computations. The parameterized layer needs to be learned to get good approximation which in- volves numerically solving an optimization problem for each layer. These optimization problems come with a computational cost that can be better controlled by using regularized inversions pre- sented earlier.
However, the approach based on parameterized inverses may lack theoretical grounding, pointed out by Bengio (2020), as we not know how close the learned inverse to the actual inverse throughout the training process. In contrast, the regularized inversion (4) is less ad hoc and clearly defined and, as we shall show the experiments, leads to competitive performance on real datasets.
In any case, the analytic formulation of the inverse gives simple insights on an approach with pa- rameterized inverses. Namely, the analytical formula suggests parameterizing the reverse layer s.t. (i) the reverse activation is defined the inverse of the activation and not any activation, (ūi) the layer uses a non-linear operation followed by a linear one instead of the usual scheme, i.e., linear operation followed by non- non-linear one.
2.2 LINEARIZED INVERSION
Earlier instances of target propagation used direct inverses of the network layers such that the target propagation update formula would read 1t-1 = fo. (v:) ) in (2). Yet, we are unaware of a success- ful implementation of TP using directly the inverses. To circumvent this issue, Lee et al. (2015) proposed the difference target propagation formula that propaarop the targets as
Ut-1 h4-1 fo. fūr (he).
If the inverses were exact, the difference target propagation formula would reduce to 0t-1 (v.). Lee et al. (2015) introduced the difference target propagation formula to mitigate the approximation error of the inverses by parameterized layers. practice, diffcrrecc-typpe target prop- agation appears be the only known successful implementation of target propagation of. The difference target propagation formula can naturally be interpreted as an approximation of the linearization used in (2), as
aware
f (v.) 1 (o. (v. he) O(I hell3),
where On Onfo. (he) denotes the Jacobian of the inverse of the layer fo.
We show in Appendix D that the first-order approximation training curves than the finiteddifference approximation. Moreover, our interpretation illuminates the "mystery" of this formula, which appeared to be critical to the success of target propagation.
propose (2) leads to slightly better
4

---

Under review as a conference paper at ICLR 2022
8
y
0p
ha
E v)
hi
he-1
he
h
hr
9
0,5
08/00
A1
Ar
def
a
do.
06 00,00,
Fig. 2: The graph of computations of target propagation is the same as the one of gradient back-propagaticn except that f needs to be computed and Jacobian of the inverses, O f are used instead gradients f in the transition operations.
3 GRADIENT BACK-PROPAGATION VERSUS TARGET PROPAGATION
Graph of computations. Gradient hack-propagation and target propagation both compute de- scent direction for the objective hand. The difference lies in the oracles computed and stored in the forward pass, while the graph of computations remains the same. To clarify this view, we reformulate target propagation terms of displacements A == Us h2 such that Eq. (1), (2) and (3) read
A, -- co (h+)),
A = =0fo. de fort E fr...., 1).
da So fo,e (h,
gnext 6n + nn... 0
Target propagation amounts then to computing a descent direction do, for the parameters 0, with graph of computations, illustrated in Fig. 2, analogous that of gradient-back. paceenttbbck-ropggattionn trated in Appendix A. The difference lies in the use of the Jacobian of the inverse
propagation illus-
0nJo, instead of OnJa, (he-1).
The implementation with the formula (2) can done in a differentiable programming frame- work, where, rather than computing the gradient of the layer, one evaluates the inverse and keep the Jacobian of the inverse. With the precise graph of computation TP and BP, we can compare their computational complexity explicitly and bound the difference of the directions they output.
Arithmetic complexity. Clearly, the space complexities of gradient back-propagation (BP) and our implementation of target propagation (TP) are the same since the Jacobians of the inverse, and the original gradients have the same size. In terms of time complexity. TP appears at first glance to introduce an important overhead since it requires the computation some inverses. However, a close inspection of the formula of the regularized inverse reveals that matrix inversion needs be computed only once for all time steps. Therefore the cost of the inversion may be amortized the length of the sequence is particularly long.
Formally, the time complexity of the Norward-backwerr pass of gradient back-propagation is essen- tially driven by matrix-vector products, i.e..
TBP
T(fe.) (an.fo.) + (do, fo.) +
Forward
Backward
B (dp + + pq) + T(²² pg),
where d is the dimension of the input zt, q is the dimension of the parameters 0n, for function we denote by T(f) the time complexity to evaluate of and we consider e.g. (he as the linear function À So (hr-)))
5

---

Under review as a conference paper at ICLR 2022
On the other hand, the time complexity of target propagation is
TTP (Vo.)+TWa))++ (00000))) ((o)
Forward
Backward
+ P(fe.c):
where P(fo,) is the cost encoding the inverse, which, our case, amounts to the cost of encoding
90 (W Win +r +rl)-1Wik AA such that regularized inverse can be computed fø,( (vz) go (v,,) #bb) Encoding, 9 comes at the cost of inverting one matrix of size p. Therefore, the time-complexity of target propagation can be estimated
TTp p² 7 p² + pq) + r(p" + pq) TBP if 2 P.
i.e., for long sequences whose length is larger than the dimension of the hidden states, the cost of TP with regularized inverses is approximately the same as the cost of BP. If a parameterized inverse was used rather than regularized inverse, the cost of encoding the inverse would correspond the cost of updating the reverse layers by, e.g., stochastic gradient descent. This update has cost similar BP. However, it unclear whether these updates get us close to the actual inverses.
Bounding the difference between target propagation and gradient pack-propaggtion. As the computational graphs of BP and TP are the same, we can bound the difference between the oracles returned by both methods. First, note that the updates of the parameters the output functions are the same since, in TP, gradients steps of the loss used to update these parameters. The difference between TP and BP lies in the updates with respect to the parameters of the transition operations. For BP, the updates computed by chain rule
Oh.
oo, l (y. go (0(:)))))
Σ So (k. 0r C(y, (b+)),
th,
where the term /h, decomposes along the time steps as Oh,/Oh, = II f (ha-1) The direction computed by TP has the same structure, namely it can be decomposed for 7h.
Sh, Onlly.co(h-)) δb.
do Σ Son (he
where 0hh/ = III.- On (h.) We can then bound the difference between the directions given by BP or TP as, for any matrix norm || || as formally stated in the following lemma.
Lemma 3.1. The difference between the oracle returned by gradiens back-propagation do, l (v. (x(T)))) and the oracle returned by target propagation can be bounded as
|10g e (v. g6 (F1rr) dell Se sup [10nSo,(hb-1) - OnJō,l (l+) 11. t=1....
where Ee Ei- an a-b--- with = sup 10n.oo.(h4--))ll =
supt= 110.fo. I1.
For regularized inverses, we have, denoting = W.A., + W Whnh,-1 b [10afa..(hr-1) (l)) [I [IV.II((Vaauu)) Va(u.) 1||- Whn + +rl1)111[Va(u)
For the two oracles to be close, we then need the preactivation u, = Wan + Whnh; + by to lie in the region of the activation function that is close to being linear s.t. Va(u,) 1. We also need (W,T, Wh, + r rI) to be close to the identity which can be the case if, e.g., r = 0 and the weight matrices Wn, were orthonormal. By initializing the weight matrices orthonormal matrices, the differences between the two oracles can closer. However, in the long term, target propagation appears give better oracles, shown in the experiments below.

---

Under review as a conference paper at ICLR 2022
TP
Train Loss
1000
1000
100
2000
2000
50
0
Iterations
x 10
Iterations
104
Iterations
x *10
100
75
75
76
Accuracy
50
50
25
25
Iterations
x10*
Iterations
x10
Iterations
x10*
Fig. 3: Temporal order problem 7' 60, Temporal Problem T = 120, Adding problem T' = 30.
Target propagation as a Gauss-Newton method? Recently target propagation has been inter- preted as approximate Gauss-Newton method, by considering that the difference target propa- gation formula approximates the linearization of the inverse, which itself is priori equal to the inverse of the gradients (Bengio, 2020; Meulemans al., 2020; 2021). Namely, provided that (h,) = I, we have
fø, (fe,)(h:-1) z h such that
On (o. (..
By composing the inverses of the gradients, we get an update similar to the one of Gauss-Newton (GN) method. Namely, recall that if n invertible functions fi....,
, fn were composed solve least square problem of the form [l o... ......o.. (I( - -yli. a Gauss-Newton update would take the ... f((=n-1)~ (I,, ---) y) where H, is defined
form (k++1) = (k) - (1 ( 22, (z,)) iteratively To (6)), 2++1 fxxx). In other words, GN and TP share the idea of composing the inverse of gradients. However, numerous differences remain: (i) in e.g. RNNs, the gradients w.rt. to the weights defined a priori not the sum of the inverses, (ii) if the real rationale of GN was used, the gradients w.rt. the loss should also be inverted, and the gradients is not the case in the usual implementation of TP Lee et al. (2015); Bengio (2020). Even if TP was approximating GN, it is unclear whether GN updates are adapted stochastic problems.
of the composition of the gradients and the inverse of this sum
to the weights should also be inverted which
EXPERIMENTS
In the following, we compare our simple target propagation approach, which we shall refer to as TP, to gradient Back-Propagation referred to as BP. We follow the experimental benchmark of Manchev & Spratling (2020) to which we add results on RNNs on CIFAR and GRUs on FashionMNIST. Additional experiments, details on the initialization and the hyper-parameter selection can be found in Appendix D.
Data. We consider two synthetic datasets generated to present training difficulties for RNNs and several real datasets consisting of scanning images pixel by pixel to classify them (Hochreiter & Schmidhuber, 1997; Le et al., 2015; Manchev & Spratling, 2020).
Temporal order problem. A sequence of length is generated using a set of randomly chosen symbols b. d). Two additional symbols X and Y are added at positions [T/10, 2T/10 and t2 E [4T/10, 5T/10J. The network must predict the correct order of appearance of X and Y out of four possible choices (XX, XY, YX, XY, YYJ.

---

Under review as a conference paper at ICLR 2022
TP --- BP
6000
Train Loss
7300
50.00
7100
7000
2000
0
7000
0.0
2.5
5.0
7.5
0.0
5.0
Iterations
10
Iterations
10
Iter attiotion
-110
It terations
100
100
100
70
70
Accuracy
50
20
25
10
2.5
6:01
7.5
900
Iterations
X10
Iterations
10
Iterations
x100
Iterations
X100
Fig. 4: Image classification pixel by pixel. From left to right: MNIST, MNIST with permuted images, CI- FARI0, PaSSIonMNIST with GRU.
Adding problem. The input consists of two sequences: one is made of randomly chosen numbers from [0. 1]. and the other one is binary sequence full of zeros except at positions t E [1,T/10] and t2 [T/10, T T/2.. The second position acts marker for the time steps t and tz. The goal of the network is to output the mean of the two random numbers of the first sequence (X,, + X22)
Image classification pixel by pixel. The inputs are images of (i) grayscale handwritten digits given in the database MNIST (LeCun & Cortes, 1998), (ii) colored objects from the database CIFARIO (Krizhevsky, 2009) or (iii) grayscale images of clothes from the database FashionM- NIST (Xiao et al., 2017). The images are scanned pixel by pixel and channel by channel for CI- FARIO, and fed to a sequential network such as a simple RNN or a GRU network (Cho et al., 2014). The inputs are then sequences of 28 x 28 784 pixels for MNIST FaShionMNIST and 32 x 32 x 3 = 3072 pixels for CIFAR with a very long-range dependency problem. We also con- sider permuting the images of MNIST by a fixed permutation before feeding them into the network, which gives potentially longer dependencies in the sequential data.
Model. In both synthetic settings, we consider randomly generated mini-batches of size 20, a sim- ple RNN with hidden states of dimension 100, and hyperbolic tangent activation. For the temporal order problem, the last layer uses a soft-max function on top of linear operation, and the loss is the cross-entropy. For the adding problem, the last layer is linear, the loss is the mean-squared error, and a sample is considered to be accurately predicted the mean squared error is less than 0.04 as done by (Manchev & Spratling, 2020).
For the classification of images with sequential networks, we consider mini batches of size 16 and a cross-entropy loss. For MNIST and CIFAR, we consider a simple RNN with hidden states of dimen- sion 100, hyperbolic tangent activation, and softmax output. For FashionMNIST. we consider a GRU network and adapted our implementation of target propagation to that case while using hidden states of dimension 100 and a softmax output.
Target propagation can tackle long sequences better than gradient pack-propagation. In Fig. 3, we observe that TP performs better than BP on the temporal ordering problem: it is able reach 100% accuracy in fewer iterations than BP for sequences of length 60 and, for sequences of length 120, it is still able to reach 100% accuracy in fewer than 40 000 iterations while BP is not. On the other hand, for the adding problem, TP performs less well than BP. The contrast in performance between the two synthetic tasks was also observed by (Manchev & Spratling, 2020) using difference target propagation with parameterized inverses. The main difference between these tasks is the different nature of the outputs, which are binary for the temporal problem and continuous for the adding problem.
In Fig. 4, we observe that TP generally performs better than BP for image classification tasks. For the MNIST dataset, it reaches around 74% accuracy after 4 10aa iterations. This phenomenon is also
8

---

Under review as a conference paper at ICLR 2022
Region parameters With convergence
10
512
250
Stepsize
Widsh
228
64
10-3
10
397
784
Regularization
Length
(6a) Conv. w.r.t. stepsize & regularization
(6b) Perf. vs width & length.
observed with permuted images, where the optimization appears smoother, and TP obtains around 86% accuracy after 4 10 iterations and is still faster than BP. On the CIFAR dataset, no algorithms appear reach significant accuracy, though TP is still faster. On the FashionMNIST dataset, where a GRU network is used, our implementation of TP performs on par with BP, which shows that our approach can be generalized to more complex networks than simple RNN.
Target propagation requires non-zero regularization term. As mentioned in Sec. by using an analytical formula to compute the inverse of the layers, we can question the interpretation of TP as a Gauss-Newton method, which would amount TP without regularization. To understand the effect of the regularization term, we computed the area under the training loss curve of TP for 400 iterations on a 1og10 grid varying step-sizes Ti and regularizations r for a fixed ?/h 10-3, The results are presented in Fig. 6a, where the smaller the area, the brighter the point and the absence of dots in the grid mean that the algorithm diverged. Fig. 6a shows that without regularization we were not able to obtain convergence of the algorithm.. Simply using the gradients of the inverse in a Gaauss-Newtor method may not directly work for RNNs. Additional modifications of the method could be added to make target propagation closer to Gauss-Newton, such inverting the layers with respect to their parameters as proposed by Bengio (2020). For now, the regularization appears to successfully handle the rationale of target propagation.
Target propagation is adapted for long sequences. In Fig. 6b, we compare the performance of BP and TP in terms of accuracy after 400 iterations on the MNIST problem for various widths deter- mined by the size of the hidden states and various lengths determined by the size of the inputs (i.e., we feed the RNN with k pixels at a time, which gives length 784/k). Fig 6b shows that TP is gen- erally appropriate for long sequences, while BP remains more efficient for short sequences. TP can then be seen as an interesting alternative for dynamical problems which involve many discretization steps as in RNNs and related architectures.
CONCLUSION
We proposed simple target propagation approach grounded in two important computational com- ponents, regularized inversion, and linearized propagation. The proposed approach also sheds light on previous insights and successful rules for target propagation. We will publicly release our code to facilitate the reproduction of the results. We have used target propagation within a stochastic gra- dient outer loop to train neural networks for a fair comparison to stochastic gradient using gradient backpropagation. Developing adaptive stochastic gradient algorithms in the spirit of Adam that lead to boosts performance when using target propagation instead of gradient backpropagation is an interesting avenue for future work. Continuous counterparts of target propagation in neural ODE spirit is also an interesting avenue for future work.
REFERENCES
Nasir Ahmad, Marcel A van Gerven, and Luca Ambrogioni. Gait-prop: A biologically plausible learning rule derived from backpropagation of error. Advances in Neural Information Processing Systems, 33, 2020.

---

Under review as a conference paper at ICLR 2022
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906, 2014.
Yoshua Bengio. Deriving differential target propagation from iterating approximate inverses. arXiv preprint arXiv:2007.15139, 2020.
Yoshua Bengio and Paolo Frasconi.
Diffusion of context and credit information in markovian mod- cls. Journal of Artificial Intelligence Research, 3:249-270, 1995.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):177-166, 1994.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.332, 2013. 2013.
Miguel Carreira- Perppnan and Weiran Wang. Distributed optimization of deeply nested systems. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics, 2014.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 2014.
Wojeiech Marian Czarnecki, Grzegorz Šwirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Sander Dalm, Nasir Ahmad, Luca Ambrogioni, and Marcel van Gerven. Scaling up learning with gait-prop. arXiv preprint arXiv:2102. 15598, 2021.
Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1-2):37-75, 2014.
Thomas Frerix, Thomas Möllenhoff, Michael Moeller, and Daniel Cremers. Proximal backpropa- gation. In Proceedings of the 6th International Conference on Learning Representations, 2018.
Ian Goodfellow. Yoshua Bengio, and Aaron Courville. Deep Learning. The MIT Press, 2016.
Akhilesh Gotmare, Valentin Thomas, Johanni Brea, and Martin Jaggi. Decoupling backpropagation using constrained optimization methods. In Credit Assignment in Deep Learning and Reinforce- ment Learning Workshop (ICML 2018 ECA), 2018.
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. In Proceedings of the 35th International Conference Machine Learning, 2018.
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Bard BBased Systems, 6(02): 107-116, 1998.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Pro- ceedings of the 34th International Conference on Machine Learning, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Quoc Le, Navdeep Jaitly, and Geoffrey Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: /504 550,00941, 2015.
10

---

Under review as a conference paper at ICLR 2022
Yann Le Cun. Learning process in asymmetric threshold network. In Disordered systems and biological organization. Springer, 1986.
Yann Le Cun, Conrad C Galland, and Geoffrey E Hinton. GEMINI: gradient estimation through matrix inversion after noise injection. Advances in neural information processing systems, pp- 141-148, 1989.
Yann LeCun and Corinna Cortes. http://yam.lecum.com. http:///////e//iiss/...oooo/ 1998.
MNIST
handwritten
digit database.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation.
In Machine Learning and Knowledge Discovery in Databases. Springer, 2015.
Mario Lezcano-Casadee and David Martunez-Rubio. Cheap orthogonal constraints in neural net- works: simple parametrization of the orthogonal and unitary group. In Proceedings of the 36th International Conference Machine Learning, 2019.
Nikolay Manchev and Michael Spratling. Target propagation in recurrent neural networks. Journal of Machine Learning Research, 21(7):1-33, 2020.
Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F. Grewe. A theoretical framework for target propagation. In Advances in Neural Information Processing Systems 33, 2020.
Alexander Meulemans, Matilde Tristany Farinha, Javier García Ordóñczz, Pau Vilimelis Aceituno, João Sacramento, and Benjamin F Grewe. Credit assignment in neural networks through deep feedback control. arXiv preprint arXiv:2106.07887, 2021.
Piotr Mirowski and Yann LeCun. Dynamic factor graphs for time series modeling. In Machine Learning and Knowledge Discovery in Databases. Springer, 2009.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. arXiv preprint 1rXiv:1211.500 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performanc deep learning library. In Advances in Neural Information Processing Systems 32. 2019.
Richard Rohwer.
The "moving targets" training algorithm. In Advances in neural information processing systems 3, 1990.
Vincent Roulet, Siddhartha Srinivasa, Dmitriy Drusvyatskiy, and Zaid Harchaoui. Iterative lin- earized control: stable algorithms and complexity guarantees. In International Conference on Machine Learning. 2019.
D. E. Rumelhart, G. F. Hinton, and R. J. Williams. Learning Internal Representations by Error Propagation. MIT Press, Cambridge, MA, USA, 1986.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural net- works. In Proceedings of the 28th International Conference on Machine Learning. 2011.
llya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum deep learning. Proceedings of the 30th International conference on machine learning, 2013.
Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein. Training neural networks without gradients: A scalable ADMM approach. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
Paul Werbos. The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting. Wiley- -ntey-Iintescceenc,, 1994.
11

---

Under review as a conference paper at ICLR 2022
Sam Wiseman, Sumit Chopra, Mare-Aurelio Ranzato, Arthur Szlam, Ruoyu Sun, Soumith Chin- tala, and Nicolas Vasilache. Training language models using target-propagation. arXiv preprint arXiv:1702.04770, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017. URL https: /ttpt/ggttttttt/////// com/
za gelaaddreeeeachh/
12


Reviews:
[{"review_id": "bEvaAbboOG2", "reviewer": "Reviewer_Dczs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents their model quite clear. Demonstration and comparison with traditional BP algorithm is illustrative. The target propagation algorithm is an interesting new directions that worth exploring. \n\nThe main downside of the paper would be experimental demonstration. \n\n-- It's good to have experiments on realistic image dataset such as cifar10, however, either the baseline as well as proposed model seems to be quite weak (their accuracy is quite low, fig 4, less than 20%). I understand the authors focused on the illustrative purpose, but that also makes it unclear how the model could generalize,  scale and apply to the realistic datasets. Since the current literature has quite a few competitive RNN models on image classification, why not directly take their models and replace the BP component to see how it works? I'm not asking for the SOTA performance, only the competitive results could be good enough to show the potential of the proposed work. \n\n-- I assume the framework is targeted on recurrent model training, perhaps the authors would make it more clear throughout the paper. Some of the places claim such as \" We have used target propagation within a stochastic gradient outer loop to train neural networks for a fair compar- ison to stochastic gradient using gradient backpropagation.\" (in the conclusion section), which might be misleading or over claimed. If in general the proposed model is flexible enough to handle all kinds of neural networks, perhaps experiments should also include convolution neural net etc for the illustration. ", "summary_of_the_paper": "The paper proposes the train the target propagation algorithm (as an alternative to gradient back-propagation algorithm) through regularized inversion. Specifically, they discussed two techniques including regularized inversion and linearize propagation, and demonstrate their effectiveness (via convergence speed and accuracy) for learning the RNN model on synthetic and image benchmarks.", "summary_of_the_review": "The paper explore an interesting alternative to BP. However, the current form didn't fully convince me its superiority over bp. I lean towards the rejection, but willing to adjust my rating based on the authors' feedbacks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YHy6Rls_bf", "reviewer": "Reviewer_NQCB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I find target propagation (TP) interesting, I am not very enthusiastic about the current paper. As detailed below, the main aspects that attract me towards TP are absent from the proposed algorithm, and its merits as a generic drop-in replacement for backpropagation are not strong enough or at least not yet well argued for.\n\nThe authors propose a new prescription to compute targets in TP. Unlike much of the recent work on TP, this prescription leads to a parameter update that requires weight transport. This limits its interest and narrows the scope of the contributions, as optimization without weight transport is of great interest not just in theoretical neuroscience but also for neuromorphic hardware design and large-scale distributed network implementations. \n\nAs noted by the authors, the proposed algorithm is also strictly costlier than backpropagation, although the relative difference in cost becomes smaller as sequence length grows. It also does not allow for decoupled updates, or at least this is not commented in the paper. Its applicability to decentralized optimization in distributed settings (another setting of potential interest for TP algorithms) is therfore not clear.\n\nFor the proposed algorithm to be interesting, it therefore has to produce descent directions that lead to faster optimization than gradients. The paper does not provide any theoretical guarantees that this is the case. The merits of the new algorithm must then be judged by the experimental results. This is a very difficult question to answer experimentally, and I did not find the results extremely convincing, but I am admittedly not an expert in sequence learning problems.\n\nI have a number of questions for the authors:\n- In page 3, below eq. 2, it is mentioned that DTP is a 0-th order approximation. Can the authors clarify this remark? In my understanding, the difference correction of DTP stems from a first-order expansion of $f$, see Appendix B of Lee et al. (2015). In what sense is it a 0-th order correction?\n\n- What are the new insights provided in section 2.2 compared to what is already used by Lee et al. to arrive at DTP (and perhaps more clearly restated in Lemma 1 of Meulemans et al. 2020)?\n\n-  I'm having trouble with eq. 4 and the text that follows. What happens when there are multiple minimizers of the objective presented in (4), for example when $a$ is the ReLU activation? How should the next unnumbered equation appearing below \"Formally, for [...] the ReLU, their inverse can be obtained analytically\" be understood in this case?\n\n- There are numerous remarks on the vanishing/exploding gradient problem and allusions to the promise of solving this problem with TP. But since the proposed parameter update relies on products of Jacobian inverses, isn't it also subject to vanishing and exploding gradient problems?\n\n- Why is the relationship between TP and Gauss-Newton optimization pointed out by Bengio (2020) and Meulemans et al. (2020) called into question due to the difficulties observed when setting the regularizer to zero? \n\n- Is GEMINI (LeCun et al., 1989), cited already in the abstract, the desired reference? To me, GEMINI is more obviously related to synthetic gradients than target propagation, and it is not usually cited as LeCun's original TP paper. Reference [1] below is perhaps more appropriate.\n\n[1] LeCun (1986) \"Learning processes in an asymmetric threshold network\"", "summary_of_the_paper": "The authors study a variant of target propagation in which targets are computed by solving a sequence of minimization problems. Instead of resorting to iterative methods the authors propose to use an analytical solution. The algorithm is investigated as a recurrent neural network learning algorithm in a number of experiments. A relationship between the proposed descent direction and the loss function gradient is developed.", "summary_of_the_review": "The algorithm loses most of the features that make target propagation attractive as an alternative to backpropagation. As an optimization algorithm, its guarantees are not sufficiently strong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QLb9l6RPlsW", "reviewer": "Reviewer_239T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The primary strengths of this manuscript are that it is easy to read, the proposed algorithm is straightforward to implement in the current computational setup, and the improvements versus gradient descent.\n\nIn my opinion, there are two main weaknesses of the manuscript.\n\n(1) The theory needs to be more fully explored and discussed.  For example, Lemma 3.1 bounds the different between the direction computed by TP and BP. First, it should be noted that this bound is quite loose in the case where the constants $a$ or $b$ are greater than 1. Second, there needs to be more complete discussion of this meaning of this Lemma, as the argument that TP is better than BP depends on the fact that TP calculates a \\textit{different} direction than BP.  I would appreciate greater follow-up and discussion in the analysis on how TP and BP differ rather than on how similar they are. Next, I find the relationship to Gauss-Newton methods quite interesting, but that section is barely sketched out and does not seem fully derived.  How close is TP to a Gauss-Newton method?  Also, I found the phrase \"In our experiments, we have not been able to get a non-diverging sequence of iterates if the regularization was set to 0, which questions the interpretation of target propagation as a Gauss-Newton method\" peculiar.  In a Gauss-Newton method, you need more samples than parameters to get the Jacobian to be well-conditioned and calculate a good direction, or it has to be regularized somehow.  The fact that you have to regularize the TP technique seems consistent with a Gauss-Newton method.  Am I missing something?  I believe that fully-fleshing out this relationship, if indeed they are close, would help provide rationale for your method.\n\n(2) The empirical evidence, in my opinion, is lacking.  There are few experiments and runs here.  When does TP outperform GD?  Why does it outperform GP?  This manuscript needs to explain more about what is different in the optimization to make it work better, and evaluate how  robust those improvements are with much more detailed and extensive experimentation, especially since the improvements are not fully explained from the theoretical side.  Additionally, there are many, many, learning algorithms designed to improve learning in RNNs.  There needs to be more comparison and discussion of these algorithms.  For example, how does this compare to the algorithm proposed in Lee et al 2015?  How does this relate to Hessian Free-Optimization (e.g., [1]), which is somewhat alluded to with the relationship to Gauss-Newton?  There must be a more complete literature search and discussion on optimizing RNNs as a whole, rather than the relatively narrow focus on exclusively TP.\n\n[1] Martens, James, and Ilya Sutskever. \"Learning recurrent neural networks with hessian-free optimization.\" ICML. 2011.", "summary_of_the_paper": "This manuscript proposes a new way of approaching optimization in RNNs based on Target Propagation.  It develops a linearized approximation to the method to make it straightforward to implement within a standard computational graph, and develops some theory about the approximations, and shows improvement compared to gradient descent in a few experiments.", "summary_of_the_review": "This manuscript proposes an interesting and potentially useful algorithm, but needs to more fully explain its theoretical properties and provide more detailed experimentation and comparisons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "48Kkr44yY1O", "reviewer": "Reviewer_Q1ay", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- The proposed scheme features approximating the inverse of the layer by a regularized inverse that facilitates the use of autograd.\n- Linearized inversion offers a natural interpretation as an approximation of difference target propagation.\n- The cost of matrix inversion can be amortized by the length of sequence.\n- There is theoretical guarantee available that bounds the difference between target propagation and backpropagation.\n\n**Weaknesses:**\n- The process seems to have large variation on training loss and accuracy compared to back propagation, which might be an indication that the scheme is less amendable to theoretical analysis.\n\n**Correctness:**\n- There is no false claims to the best of my knowledge\n\n**Clarity:**\n- The paper is clear and structured\n- A typo in the paragraph under equation (2): \"allow us tp interpret\" ---> \"allow us to interpret\"\n\n**Additional Comments:**\n- Is there any convergence result possible for the proposed target propagation scheme given that the backward updates are more tractable compared to the original version? For example, there is a convergence analysis for feedback alignment by Song et al.. Anything similar possible for target propagation?", "summary_of_the_paper": "The authors provide a modified version of target propagation that is easily implementable via autograd system. This paper also compares the computation complexity of the algorithm with backpropagation and shows empirically that the algorithm yields good performance on recurrent neural networks with long sequences.", "summary_of_the_review": "The target propagation scheme proposed in this paper look novel to me. Even though the numerical result is a bit confusing, the algorithm has natural interpretation and more computationally tractable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach to target propagation that performs well when used in RNNs on sequence modelling. The paper falls into something of an uncanny valley, where it is different enough from the original TP to lose some of its motivation ("biological plausibility"), and is now directly competing with backprop. Claims about outperforming backprop require EXTREMELY thorough and rigorous experimental evidence. Without meaning to cast any doubt on the authors work, there have simply been a lot of papers over the years that saw some improvements over backprop in some setting, that have not generalised or even been reproducible.
Decision: Reject