BP($\mathbf{\lambda}$): Online Learning via Synthetic Gradients
Authors: Joseph Oliver Pemberton
Rui Ponte Costa
Venue: Accepted by TMLR ()

Abstract: Training recurrent neural networks typically relies on backpropagation through time (BPTT). BPTT depends on forward and backward passes to be completed, rendering the network locked to these computations before loss gradients are available. Recently, Jaderberg et al. proposed synthetic gradients to alleviate the need for full BPTT. In their implementation synthetic gradients are learned through a mixture of backpropagated gradients and bootstrapped synthetic gradients, analogous to the temporal difference (TD) algorithm in Reinforcement Learning (RL). However, as in TD learning, heavy use of bootstrapping can result in bias which leads to poor synthetic gradient estimates. Inspired by the accumulate $\mathrm{TD}(\lambda)$ in RL, we propose a fully online method for learning synthetic gradients which avoids the use of BPTT altogether: \emph{accumulate} $BP(\lambda)$. As in accumulate $\mathrm{TD}(\lambda)$, we show analytically that {accumulate~$\mathrm{BP}(\lambda)$} can control the level of bias by using a mixture of temporal difference errors and recursively defined eligibility traces. We next demonstrate empirically that our model outperforms the original implementation for learning synthetic gradients in a variety of tasks, and is particularly suited for capturing longer timescales. Finally, building on recent work we reflect on accumulate $\mathrm{BP}(\lambda)$ as a principle for learning in biological circuits. In summary, inspired by RL principles we introduce an algorithm capable of bias-free online learning via synthetic gradients.

Published in Transactions on Machine Learning Research (04/2024)
())) Online Learning via Synthetic Gradients
Joseph Pemberton
jo..pemberton@Sbriitall.cc uk
Compulational Newroscience Unit, Faculty of Engineering, University of Bristol, United Kingdom Centre for Neural Circuits and Behaviour, Department of Physiology, Anatomy and Genetics, Medical Sciences Di- vision, University of Oxford, United Kingdom
Rui Ponte Costa
rui.costa Edpag. Sodpog.ox.da ac. uk Centre for Newral Circuils and Behaviour, Department of Physiology. Anatomy and Genelies, Medical Sciences Division, University of Ouford, United Kingdom
Compulational Newroscience Unit, Faculty of Engineering, University of Bristol, United Kingdom
Reviewed on OpenReview: https: // openreview. net/ forum? id= dd33VgouAfqh
Abstract
Training recurrent neural networks typically relies on backpropagation through time (BPTT). BPTT depends forward and backward passes to be completed, rendering the network locked to these computations before loss gradients
available. Recently, [ader- berg et al. proposed synthetic gradients to alleviate the need for full BPTT. In their imple- mentation synthetic gradients are learned through a mixture of backpropagated gradients and bootstrapped synthetic gradients, analogous to the temporal difference (TD) algorithm Reinforcement Learning (RL). However, as in TD learning, heavy use of bootstrapping can result in bias which leads poor synthetic gradient estimates. Inspired by the ac- cumulate TD(A) in RL, we propose a fully online method for learning synthetic gradients which avoids the use of BPTT altogether: accumulate (P))). As in accumulate TD(A), we show analytically that accumulate (P()) can control the level of bias by using mixture of temporal difference errors and recursively defined eligibility traces. We next demonstrate empirically that our model outperforms the original implementation for learning synthetic gradients in a variety of tasks, and particularly suited for capturing longer timescales. Finally, building on recent work we reflect on accumulate BP(A) as a principle for learning in biological circuits. In summary, inspired by RL principles introduce algorithm capable of free online learning via synthetic gradients.
1 Introduction
A common approach for solving temporal tasks is to use recurrent neural networks (RNNs), which with the right parameters can effectively integrate and maintain information over time. The temporal distance between inputs and subsequent task loss, however, can make optimising these parameters challenging. The backpropagation through time (BPTT) algorithm the classical solution to this problem that applied once the task complete and all task losses are propagated backwards in time through the preceding chain of compulation. Exact loss gradients are thereby derived and are used to guide updates to the network parameters.
However, BPTT can be undesirably expensive to perform, with its memory and computational requirements scaling intractably with the task duration. Moreover, the gradients can only be obtained after the RNN forward and backward passes have been completed. This makes the network parameters effectively locked until those computations are carried out. One common solution alleviate these issues is apply truncated BPTT, where error gradients are only backpropagatod within fixed truncation windows, but this approach can limit the network's ability to capture long-range temporal dependencies.

---

Published in Transactions on Machine Learning Research (04/2024)
b
unavailable BPTT gradient
a ahu
Lt
LT
Dt
lyt
ht
Et
hv
RNN
Gt
G+tl
G
8
8141
||G. - -Gll
e1
Ctt1
ht
time
synthesiser
6: G/+1
Oht+1
ahe
Ge
et =
Oh4-1
ahe
aG
00,
20 81 Tel
Figure 1: Schematic of recurrent neural network (RNN) which learns via synthetic gradients. (a) External input is provided to the RNN which has hidden state h2- Due to recurrency this state will affect the task loss at the current timestep L, and future timesteps L not yet seen. A distinct synthesiser network receives he as input and estimates its future loss gradient G, Bl.s which provided to the RNN The
for learning. The synthesiser learns to mimic target gradient G.. How G. is defined and learned is the focus of this paper. (b) An illustration of the accumulate (P(A) algorithm for learning synthetic gradients in an unrolled version of the network. Current activity h2 must be correctly associated to the later task loss Lr. Here the parameters of the synthesiser are updated via a mixture of temporal difference errors 5 (red) and eligibility traces e (green). As in accumulate TD(A) in RI, 5 is computed online using bootstrapping whilst e propagates forwards with a decay component À with X S 1. Together, they approximate the true loss gradient. In contrast the original synthetic gradient algorithm by Jaderberg et al. (2117). our model does not require BPTT.
One proposed method which avoids the need for many-step BPTT whilst still capturing long-range de- pendencies is to apply synthetic gradients (Jaderberg et al.) 2017 Czarnocki et al. 2017). In this method gradients from future errors are predicted by separate work, "synthesiser", given the current RNN activity (Fig. [ Synthetic gradients enable the network to model long-range dependencies on future errors whilst avoiding the waiting time imposed by BPTT. The independence of memory and computational com- plexity with respect to the total task length makes synthetic gradients an attractive alternative compared to BPTT (Marschall et al. 2020). Recently, these properties have led neuroscientists speculate that synthetic gradients are computed at the systems-leve in the brain, explaining range of experimental observations
Marschall et al. 2019 Pemberton et al. 2021 Boven et al. 2023
Despite their promise, the full potential of approximating BPTT with synthetic gradients has not yet been realised. In particular, it is not yet clear what are the optimal conditions for learning synthetic gradients. In its original implementation, Jaaerberg et al. use synthetic gradients alongside truncated BPTT, and define the synthesiser target as a mixture of backpropagated gradients with its own predicted (future) gradient. That is, the synthesiser uses its own estimations bootstrapping - for learning. As the original authors note, this is highly reminiscent of temporal difference (TD) algorithms used in Reinforcement Learning (RL) which use bootstrapping for estimating the future return (Sutton & Barto 2018). Indeed, in their supplementary material Jaderberg et al.) extend this analogy and introduce the notion of the X-weighted synthetic gradient, which is analogous to the X-return in RL. However, X-weighted synthetic gradients were only presented conceptually and remained unclear whether they would be of practical benefits as they still require BPTT.
In this study, inspired by established RL theory, we make conceptual and experimental advancements on X-weighted synthetic gradients. In particular, we propose an algorithm for learning synthetic gradients, accumulate PP)), which mirrors the accumulate TD(A) algorithm in RL (Van Seijen al. 2016). Just.
2

---

Published in Transactions on Machine Learning Research (04/2024)
how accumulate (D(A) provides online solution to learning the A-return in RL, show that accumulate (P(A) provides an online solution to learning X-weighted synthetic gradients. The algorithm uses forward- propagating eligibility traces and has the advantage of not requiring (even truncated) BPTT at all. Moreover, we demonstrate that accumulate (P()) can alleviate the bias involved in directly learning bootstrapped estimations as suffered in the original implementation.
We now provide a brief background into the application of synthetic gradients for RNN learning. We then introduce the accumulate BP()) algorithm and demonstrate both analytically and empirically can it alleviates the problem of bias suffered in the original implementation. Next, touch upon accumulate (A( as mechanism for learning in biological circuits. Finally, we discuss the limitations and conclusions of our work.
2 Background
2.1 Synthetic gradients for supervised learning
Consider an RNN with free parameters Vi performing a task of sequence length T (which may be arbitrarily
long). At time RNN dynamics follow h; = f(x,, ht-1; V), where h is the RNN hidden state, x is the input, and f is the RNN computation. Let L, [(e. Me) denote the loss at time t, where de is the RNN-dependent prediction and ye is the desired target. Tet T2x Eierst T., denote the total loss strictly after timestep t.
During training, wish to update v to minimise all losses from timestep & onwards. Using gradient descent this is achieved with v = v - 04
. "iiser
for some RNN learning rate 1/. We can write this gradient as
0 EiSTST I.
0(L. + Lsze)
(1)
0y
0% ah
aL.
OV + OU
(2)
@h, Thzt Oh,
+
(3)
Oh,
Oh,
Whilst the terms 8L, and above are relatively easy to compute and available at timestep t, The can Th, 24
OLas
ōL 0, and future errors are present challenges. Without BPTT, this term simply taken as zero, Sh. effectively ignored. With BPTT, this term is only computed after the forward pass and the corresponding backward pass so that all future errors are observed and appropriately backpropagated. This has memory and computational complexity which scales with T, and thereby relies on arbitrarily long waits before the loss gradient is available.
The aim of synthetic gradients is to provide an immediate prediction G, of the future loss gradient, G G::= == 01.se [Jaderberg al. 2017.. We use notation G, both to represent "gradient" " but also to highlight its -hh
resemblance to the return in RL. Note that this gradient is a vector of the same size as h. As in the original implementation,
consider G2 to be computation of the current RNN state with separate "synthesiser" network: G, = g(h,; 0), where y denotes the synthesiser compulation which depends on its free parameters 0. An approximation for the loss gradient with respect to the RNN parameters can then be written as
2 EiSrST Lr OV
@T. Oh,
G.
The 2V
(4)
This is available timestep & and removes the dependence of the memory and computational complexity
T as in Eq. 3
3

---

Published in Transactions on Machine Learning Research (04/2024)
2.2 Learning synthetic gradients
How the synthesiser parameters 0 should be learned remains relatively unexplored and is the focus of this paper. The problem can be stated as trying to minimise the synthesiser loss function ()( defined as
2 c -G1
(0)= == En,p
(5)
Σ
P (ha) llve y(hrs 0)1
(6)
2
where P is the probability distribution over RNN hidden states, G. is the target synthetic gradient for state he, and Il-1lz denotes the Euclidean norm. By taking samples of hidden states he and applying the chain rule, the stochastic updates for 0 can be writlen as
Og(he; 0.)
Dc+1 O2 a (e. g(hp; 0.))
(7)
00,
where a is the synthesiser learning rale. Note the transpose operation T since all terms in Equation are vectors.
Idcally G2 is the true error gradient, G2 G2, but this requires full BPTT which is exactly what synthetic gradients are used to avoid. In its original Tormulation, [Jadeberg et al. instead propose a larget which is based on mixture of backpropagated error gradients within fixed window and bootstrapped synthetic gradient to incorporate errors beyond. The simplest version of this, which only uses one-step BPTT, only incorporates the error at the next timestep and relies on a bootstrap prediction for all limesteps onwards. We denote this target G(().
G = ah1+1 + +nG++1 Ohyt1 Ohe
dh:
where, inspired by RL, y E 10,1] is the gradicnt discount factor which if y 1 scales down later error gradients. Note that in its original implementation 7 1, but in our sinulations we find it importanl to set < 1 (see Appendix section
Notably, Equation resembles the bootstrapped target involved in the TD algorithm in RL Sutton & Barto (2018). In particular, G(!) can be considered analogous to the one-step return at state S, defined as Ret1+ (V(+++)), where R441 is the reward, S4+1 is the subsequent state, and V is the (bootstrapped) value function.
Table 1: Summary of terms used in value estimation in Reinforcement Learning (RL) and synthetic gradient (SG) estimation in supervised learning. For RL RG denotes the reward at timestep t, oe denotes the feature-based representation of the environmental state, and V is the value function. See main text and Van Seijen et al. (2066) for further details of the terms.
Reinforcement Learning
Synthetic Gradients
G
Σ9217" 1 BRt4
Σx>1 y-- Bh.
OL
Ge
V(ou:))
g(he:0) 0)
((1))
T1
BetT + +7" G
Ohi ΣT=1 -- 01112 + +7 +TT
Sh.
ah.
G
(1 - A) n=
1n Am (1) + X2-
G
(1 A) - HH x--1C(1) A -(
2.3 n-step synthetic gradients
In practice, in its original implementation Jaderberg et al. primarily consider applying synthetic gradients alongside truncated BPTT for truncation size 1.
4

---

Published in Transactions on Machine Learning Research (04/2024)
In this case, the left side term in Equation S can be extended to incorporate loss gradients backpropagated
within the n timesteps of the truncation. The synthesiser target then set as G. = G!")) where G!") the n-slep synthelic gradient defined as
G
--- E
ÖL. oh, + + Grtn
Uhu+n
(9)
dh,
BETEE+n
which is analogous to the n-step return RL. Importantly, in practice this scheme uses truncated BPTT to both define the synthesiser target and the error gradient used to update the RNN. Specifically, Jaderberg jet al. apply the backpropagated gradients to directly learn V well as 0, with the synthetic gradients themselves only applied at the end of cach truncation window (see original paper for details).
Just as RI, increasing the truncation size n provides a target which assigns more weight to the observations than the bootstrapped term, thus reducing its bias and potentially leading to better synthesiser learning. On the other hand, Equation g requires n-step BPTT and thereby enforces undesirable waiting time and complexity for large n.
3 Model and analytical results
In this study we formulate a learning algorithm accumulate BP which has the advantage of reduced bias compared to the original n-step implementation. Furthermore, the algorithm can be implemented each timestep and does not require BPTT at all.
We first define X-weighted synthetic gradients. We highlight that our definition is similar, but not the same, as that first introduced in Jaderberg ct al. (2017). Specifically, our definition incorporates loss gradients for losses strictly after the current limestep and can therefore naturally be used the context of learning the synthesiser.
3.1 X-weighted synthetic gradient
Let be such that 0 s A < We define the X-weighted synthetic gradient as
G =1 - A)
n ((n)
(10)
n=1
T-1-1
(1 A)
A G + +XT-1-1G
(11)
==1
which is analogous the A-return in RL.
Note the distinction in notation between the X-weighted synthetic gradient G. and the n-step synthetic gradient G("), Moreover, note that Equation 11 is similar but distinct to the recursive definition as proposed in the original paper (see Appendix section B In particular, whilst Jaderberg et al. also incorporate the loss gradient at the current timestep in their definition, Equation II only considers strictly future losses. Since the synthesiser itself is optimised to produce future error gradients (Equation 4h this enables the X-weighted synthetic gradient to be directly used as synthesiser target.
As in RL, a higher choice of À results in stronger weighting of observed gradients compared to the boot- strapped terms. For example, whilst G. c(")) is just the onc-step synthetic gradient which relies strongly the bootstrapped prediction (cf. Equation 8 G G. is the true (unbiased) gradient obtained via full BPTT.
We also define the interim X-weighted synthetic gradient G GA as
G =1 (1
Σ
(n)
+ H H- k--k-1 G (k)
(12)
n=1
5

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1 RNN learning with accumulate BP(A). Updates RNN parameters using estimated
gradients provided by synthesiser function 9.
Input: Vo: Oo, (e) 7,, a, 7, A
V < Vo
finit. RNN parameters]
0 4-+ 8a
finit. synthesiser parameters)
h, On. f-0 0
{init. RNN state, Jacobian, and clig. trace]
for / = 1 to T do
E - exdne + 0g(h:9]
{update eligibility trace}
08
- fii, h; V) L t [(h', y1) Or.- + aL
fcompute next hidden state and task loss
0n - 8h
an
0h Oy x- ah
(compute local gradients)
200
0 - Or + +ggh' hi; 0) 0n gh;
fcompute synthesiser TD error)
AO t adT T e
{update synthesiser paramcters {update RNN parameters] {update RNN hidden statc}
AV +- 110 + 0)) By
h h
end for
Unlike Equation this is available at time ("horizon") with k < H <T
Table provides an overview of the terms defined along with their respective counterparts RL.
3.2
Offline X-SG algorithm
We define the offline X-SG algorithm to learn which is analogous to the offiine A-return algorithm in RL but for synthetic gradients (SG), by taking G, Gp in Equation 7 It is offline in the sense that it requires the completion of the sequence at timestep T before updates are possible.
3.3 Online X-SG algorithm
We define the online A-SG algorithm to learn 8, which is analogous to the online A-return algorithm in RL but for synthetic gradients. the current limestep I, the algorithum updates based on its prediction over all prior RNN hidden states h2 from & up k =t In particular, timestep the algorithm applies many weight updates by iterating over i according to
T Og(hs: (/11)
= 0% + (G g(hn: (11x;0%=1)
(13)
00,
where 0g = fin is the initialisation weight and subsequent initial weight vectors are inherited from the previous timestep, 0++1 = fÇ. This latter quantity the final weight vector at time defines 0, (without superseript).
The information used for the weight updates in Equation 13 is available at the current timestep and the algorithm is therefore online. Moreover, in RL, the online X-SG algorithm produces weight updates similar to the offline X-SG algorithm. In particular, at the end of the sequence with the horizon H = T note that G. and G² are the same. However, to store the previous hidden states and iteratively apply the online X-SG algorithm requires undesirable computational cost. In particular, the 1 1+1 2 + T operations in Equation 13 result in computational complexity which scales intractably with 7
3.4 Accumulate BP(A)
In this study we propose the accumulate (A) algorithm which is directly inspired by the accumulate TD(A) algorithm in RL (Van Seijen et al.] 2016). Like accumulate TD(A), the motivation for accumulate (A) is enable relatively cheap, online computations whilst alleviating the problem of bias which comes from bootstrapping using eligibilily traces (Fig. [p).

---

Published in Transactions on Machine Learning Research (04/2024)
In accumulate (P()), the weight update timestep t is defined by
Oc+1 = 0, + od0g ex
(14)
Where & is the temporal difference error at timestep t
6, 0Lu+1
Th1-1 g(hbi Oh,
+ (00)))1100)
(15)
dh,
and et is the eligibility trace of at time t
ah, et = 7d
0g(he;0) 0z)
ct-1 +
(16)
-h
00,
with en defined as the zero vector, ep =
Point of clarity. Note that there is some abuse of notation with respect to the matrix multiplication operations defined in Equations [4 and I6 If i outs are the sizes of the input and output dimension of 0, respectively, then the cligibility trace e is three-dimensional vector of shape (lh), outs, ing), where |h] is size of h. To compute the matrix product Ae, for vector A of shape (r, [hl) we concatenate the latter two dimensions of e, so that is of shape (lh), outp x ino) and once the product is computed reshape it as (r, oute, ing). Note that r (as in Equation 14 then the first dimension is removed.
The core idea behind accumulate BP(A) is that these eligibility traces keep track of the weights which have contributed to previous error-gradient predictions, where the rate at which this contribution fades over time is Xy. These traces then inform the synthesiser which weights are more liable to change once temporal difference error occurs (Equation 15). Importantly, accumulate BP(A) applies forward-view (online) method of learning which operates limestep by timestep, in stark contrast to the (backward-view) offline X-SG algorithm for which the task sequence must first be completed before updates can occur.
The main analytical result this paper is that accumulate BP(A) provides an approximation to the online X-SG algorithm. This theorem uses the term A defined as
A (G g(hx;00)) 0g(h;;0)) 00;
(17)
where G is the A-weighted synthetic gradicnt which uses the initial weight vector Gg for all synthetic gradient estimations.
Theorem 3.1. Let fy be the initial weight vector, 8 be the weight vector at time t computed by accumulate BP(A), and 02 be the weight vector al time i computed by the online X-SG algorithm. Furthermore, assume that D Ai does not contain any zero-elements. Then, for all time steps t:
0.P
0/111 112
+ 0
as a a-> 0
8MP 80112
Proof: The full proof can be found in Appendix section A In general, the structure of the proof closely follows that provided in the analogous RL paradigm for accumulate T ((A) (Van Seijen et al. 2016 D.
Accumulate BP()) thus provides an online method for learning the X-weighted synthetic gradient which, analogous to accumulate TDA), avoids the memory and computational requirements of the online X-SG algorithm. For example, when eligibility traces are unused and X = 0, i.e. accumulate B PPD),, the synthesiser learns with the one-step synthetic gradient as its target, G, = ((1) and arrives at the original implementation with truncation size n (Jaderberg et al. 2017). When = 1, i.e. accumulate PI),,, for an appropriately small learning rate a the synthesiser learns with true BPTT gradient as target, G, = G.- Importantly, Equations 15 and only consider gradients between variables of at most 1 timestep apart. In this respect, at least in the conventional sense, there is BPTT.
7

---

Published in Transactions on Machine Learning Research (04/2024)
1=
(aderberg al.)
0.5
==1
alignment to true gradient
1.0
alignment to true gradient
1.0
0.8
A=
0.6
A=
A
(=7-1
0.4
0.75
(=7-5
A= 1
t= T 7
0.2
0.0
100
50
T-2 T-3
T-5 T-6 T-7 T-9
epoch
timestep
Figure 2: (A) derives true BPTT gradients over multiple timesteps. In this toy paradigm, imput is only provided at timestep 1 and the task target is only available at the end of the task at time = 10. (a) Alignment between synthetic gradients and true gradients for a fixed RNN model across different timesteps within the task, where the synthetic gradients are learned using (accumulate) BP()). Aligmment is defined using the cosine similarity metric. (b) The average alignment over the last 10% epochs in a across all timesteps.
4 Experiments
Next, we tested empirically the ability of accumulate (P()), which we henceforth simply call BP()), to produce good synthetic gradients and can drive effective RNN parameter updates. In all experiments we take the synthesiscr computation g simply as linear function of the RNN hidden state, g(he 0) = Ah.-
4.1 Approximating true error gradients a toy task
We first analyse the aligmment of which we use quantify the bias in synthesiser predictions. For this we consider a toy task in which fixed (randomly connected) lincar RNN receives static input I1 at timestep and mull input onwards, Ha = 0 for <> 1. To test the ability of (P to transfer error information across time the error is only defined at the last timestep Lt, where LT is the mean-squared error (MSE) between a two-dimensional target yr and linear readout of the final hidden activity lt.. We use a task length of T = 10 timesteps. Note that since the network linear and the loss is a function of the MSE, the lincar synthesiser should in principle be able to learn perfectly model the true BPTT gradient (Czarnecki et al. 2017).
synthetic gradients and true gradients derived by full BPTT,
As expected, we find that high À improves the alignment of synthetic gradients and true gradients compared to the À case (i.e. BP(0) in Jaderberg et al. (2017); Fig. 22). Specifically, these results show, predicted, that the heavy reliance on bootstrapping of BP(O) means that loss gradients near the end of the sequence must first be faithfully captured before earlier timesteps can be learned. When eligibility traces are applied in BP(1), however, the over-reliance on the bootstrapped estimate is drastically reduced and the synthesiser can very quickly learn faithful predictions across all timesteps. Indeed, we observe that high À avoids the deterioration of synthetic gradient quality at earlier timesteps as suffered by BP(0) (Fig. 2p). We also observe that (A) often outperforms n-step synthetic gradients (Fig. 6)
Next, to verify that (P(A) is actually beneficial for RNN learning, followed the scheme of Jaderberg al. (2017) and applied RNN weight updates together with synthesiser weight updates (Algorithm 1 To ensure that the RNN needs to learn temporal association (as opposed to a fixed readout), in this case we consider 3 input/targe pairs and consider large sequence lengths T; we also now apply a tanh non-linearity to bound RNN activity.
Consistent with our results for the case of fixed RNNs, we observe that BP(A) gives rise to better predictions for high À when the RNN is learning (Fig. 3a). Notably, however, the alignment is weaker for plastic RNNs
8

---

Published in Transactions on Machine Learning Research (04/2024)
fixed RNN
plastic RNN
alinment gradient
1.0
0.8
error (MSE)
10
BP(A)
online A-retum
10-5
0.05
10-7
T
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.2
0.4
0.6
0.8
b
T= 10
T 30
== 60
1.00
BPIO)
(jderberg et al.) BP(0.5) BP(1)
error (MSE)
0.75
0.50
0.25
0.00
100
200
100 200
100
epoch
Figure 3: (A drives better RNN learning in a toy task. (a) Average cosine similarity bet ween synthetic gradients and true gradients for fixed (left) and plastic (right) RNNs. Cosine similarity for plastic RNNS is taken over the first 5 training epochs, since this initial period is the key stage of learning (i.e. before the task is perfected).
(b) Learning curves of RNNs which are updated using synthetic gradients derived by BP(X) over different sequence lengths T. (c) Effect of learning rate a (cf. Equation 14 on task error (averaged over last 10 epochs of training) for plastic RNNs using synthetic gradients derived by BP(A) and the online X-SG algorithm (X = T 10); this can be compared to Figure 2 in Van Seijen et. al. (2016) which displays analagous results in the RL paradigm. Here standard stochastic gradient descent is used (all other main experiments use Adam). Results show average (with SEM) over different initial conditions.
when compared to fixed RNNs. This is because the challenge is now harder for the synthesiser, since changes to the RNN will affect its error gradients and therefore lead to a moving synthesiser target. Nonetheless, we find that BA(A) improves RNN learning even in the presence of relatively long temporal credit assignment (Fig. 3D). Next, we contrasted the ability of both BP(X) and n-step methods to achieve near-zero error. Our results show that BP(1) is able to solve tasks more than double the length of those solved by the next best model (Table 2 and Fig.
Finally, we explored the effect of learning rate with BP(A). Indeed, Theorem 3.1 only applies as the step-size a approaches zero, and empirically TD(A) is known to diverge observe similar results in our experiments, and note that BP(A) is only applicable with modest step-sizes (Fig. 3c). This is true over range of values for X, though, consistent with TD(A), we observe that mid-range À values more robust to the learning rate (Fig. g e.g. compare X = 0.75 with X =).
too large Van Seijen et al. (2016). We
4.2 Sequential MNIST task
To test the ability of BP(X) to generalisc to non-trivial tasks, we now consider the soquential MNIST task [Le et al., 2015). In this lask the RNN is provided with a row-by-row representation of an MNIST image which it must classify at the end (Fig. [ That is, as in the toy task above, the task loss only defined at the final timestep and must be effectively associated prior inputs. Since this harder task we now use non-linear LSTM units in the RNN which are better placed for these temporal tasks (Hochreiter & Schmidhuber 1997
9

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Overview of model performance for sequential MNIST and copy-repeat tasks. Results for toy and copy-repeat tasks shows the average task sequence length solved by the models (sce text). Results for the sequential MNIST task show the average test accuracy percentage after training, Values denote average over 5 different initial conditions.
BPTT
BPTT
+
SG
no BPTT
n=2
n=3
n=4
n=5
n=2
n=3
n=4
n=5
n=
BP( BP(0.5)
BP(1)
toy task
16
10
20
8
36
38
24
20
4
16
32
90
sequential MNIST
73.4
78.4
82.0
87.1
78.2
83.0
86.5
90.4
64.1
69.6
76.6
90.6
copy-repeat
9.0
9.0
12.0
15.0
12.0
9.0
15.0
23.0
8.2
9.0
15.8
29.0
no BPTT madels
BPTT models
100%
100%
ttime
75%
75%
Secoree 1980
accurres
50%
no 8PTT
50%
SPTT [n=2]
BPTT + (n=2)
output dass
BP10) (jderberg et al.) BP12)
BPTT (n=3)
BPTT + 5G In-3]
3
9 7 3
BPTT In
BPTT + 56 (n=5)
25%
25%
examples
opech
opoch
Figure 4: Performance of (A) in sequential MNIST task. (a) Schematic of task. Rows of an MNIST image are fed sequentially as input and the model must classify the digit at the end. (b) Validation accuracy during training for (A) models. (c) Validation accuracy during training for models which learn synthetic gradients (SG) with n-step truncated BPTT in original implementation (Jaderberg et al. (017): final performance of BP(1) (as in (b); dotted green) is given for reference. Results show mean performance over different initial conditions with shaded areas representing standard error of the mean.
Our results show that the use of (X significantly improves on standard no-BPTT model, i.e. with 0L.t in Eq. 3 (Figs. 4b and 9 Consistent with results from the toy task we find that high X ah. produces laster rales of learning and higher performance. For example, BP(1) achieves error less than third of that achieved by the eligibility trace free model, BP(0) (~ 10% vs 30%). Moreover, BP(1) generally outperforms the models considered by JJaerberg et al. (017 which rely on truncated BPTT and the n-step synthetic gradient with n (Fig. 46 and Table 2
4.3 ropeet task
Finally, we consider task with more complex and longer temporal dependencies the copy-repeat task Graves et al. 2014).. In this task the model receives input start delimiter followed by an 8-dinensional binary sequence of length N and repeat character R. The model must then output the sequence R times before finishing with a stop character. The total sequence length is then T = N x (R + 1) + 3. We follow the procedure as set out in Jaderberg et al. (2017) and deem a sequence length solved if the average error is less than 0.15 bits. Once solved we increment. N and R alternatively.
We again observe that BP(1) provides the best BPTT-free solution in solving large sequence lengths and outperforms the n-step synthetic gradient methods (Fig. 5 and Table. 2 We do however note that our implementation of these n-step methods fails to reach the performance thresholds as presented in the original
10

---

Published in Transactions on Machine Learning Research (04/2024)
a
b
no BPTT models
BPTT models
30
30
sequence length solved
no BPTT
sequence length solved
25
BPTT (n=2)
20
BP(0)
(Jaderberg et al.)
20
BPTT (n-3)
BPTT (n=4)
15-
8P(0.5)
BP(1)
15
BPTT (n=5)
BPTT + SG (n=2)
10
BPTT + SG (n=3)
BPTT + SG (n=4)
BPTT + SG (n=S)
15
10
15
# training examples (millions)
# training exa examples (millions)
Figure 5: Performance of BP(A) copy-repeat task. (a) Maximum sequence length solved for (P()) models. A sequence length is considered as solved the model achieves an average of 0.15 bits error for given length. (b) Maximum sequence length solved for models with n-step synthetic gradient (SG) learning methods (Jaderberg al. 2017 best task performance of (P(1) (as in (a); dotted green) is shown for reference. See also Table for more details. Results show mean performance over 5 different initial conditions with shaded areas representing standard error of the mean.
paper (Jaderberg el al. 2017 but that they are consistent with those reported in Bellec et al. 20199) This is perhaps due to the more modest hyperparameter choices (e.g. less hidden units) used in our models.
5 Relevance for learning in biological networks
Table 3: Complexity of the (A) algorithm against backpropagation through time (BPTT) and real-time recurrent learning (RTRL) based algorithms for fully connected recurrent neural networks (RNNs). |h| is the hidden size of the RNN; T is the task length; n is the imposed truncation size. Time complexity is reported per timestep (i.e. the average complexity over the sequence of timesteps) and per "pass", where pass refers to the period over which the gradient is computed (e.g. over each truncation for truncated BPTT, or over each timestep for RTRL and BP())). | denotes the number parameters in the RNN, whilst (0] denoles the number of parameters in the synthesiser.
Time (per timestep)
Time
Algorithm
Memory
(per pass)
Full BPTT Truncated BPTT
|4|T [//
h//2
h
|
[ 12
Synthetic gradients (original)
[h/n
[h]
[h 22
(P (ours)
[b][0]
[M|3 0
| [x[210)
RTRL
[|]]
[A]2 /
The BP(X) algorithm is of potential interest neuroscience. Unlike BPTT which is considered biologically implausible (Lillicrap & Santoro 2019 Prince et al.) 2021), BP(A) is fully online and avoids the need to store intermediate activations (and complex gradient caleulations back in time) across arbitrary mumber of timesteps. Moreover, the application of synthetic gradients has relatively cheap computational and memory costs when compared to other online learning algorithms (Marschall el. al.) 2020). Perhaps most interestingly, BP(A) employs a combination of retrospective and prospective learning, each of which are thought to take place the brain (Namboodiri Stuber 2021). Specifically, whilst the main network learns via prospective learning signals (i.c. synthetic gradients), the synthesiser itself learns in retrospective manner by using forward-propagating eligibility traces.
11

---

Published in Transactions on Machine Learning Research (04/2024)
One possible candidate for the expression of synthetic gradients in the nervous system are neuromodulators. For example, dopaminergic neurons are known to encode expectation of future reward or error signals [Hollerman & Schultz) (998) and have been observed to play an important role in mediating synaptic plasticity (Yagishita et al.) 2014; Gerstner et al.) 2018). Furthermore, as required for synthetic gradient vectors; there is increasing evidence for signilicant heterogeneity in the dopaminergie population in areas such as the ventral tegmental area, both in its varioty of encoded signals and the targeted downstream circuits (Lerner al., 2015 Beier et al. 2015 Avvisati et al., 2022). Each signal may thus reflect a predicted gradient with respect to the target cortical cell or cell ensemble.
More recently, it has also been suggested that a particular subcortical structure - the cercbellum - predicts cortical error gradients via the cortico-cerebellar loop (Pemberton et al. 2021 Boven et al. 2023 The cerebellum would thus act as a synthesiser for the brain, and it is suggested that a bootstrapped learning strategy may be line with experimentally observations (Ohmae S. 2019 Kawato et al. 2021). (A( offers an extra degree of biological plausibility to these studics by removing the need for BPTT at all. Morcover, the algorithm makes specific predictions regarding the need for eligibility Lraces (cf. Eq. 16) al key learning
sites such as the cerebellar parallel fibres (Kawato et al. 2011).
6 Limitations
Like the originally proposed algorithm for synthetic gradients, BP(A) makes the assumption that future error gradients can be modeled as function of the current activity in the task-performing network. The algorithm may therefore suffer when the mapping between activity and future errors becomes less predictable, for example when the task involves stochastic inputs. Indeed, we find that whilst (P works well for tasks with fairly structured temporal structure as in the experiments presented, the algorithm can fall short in tasks with little or no temporal correlation between lask inputs. For example, we failed to observe meaningful gains with (A in the LISTOpS task which employs randomly generated inputs (Tay et al., 2020). On a related note, it may be that in the case of variable, non-deterministic settings BP(A) may be prone to a high variance in the synthesiser target gradient compared to the more biased BP(O), resulting in potential instability during learning as can occur, for example, in Monte Carlo algorithms in RL. However, how the bias-variance tradeoff in RL exactly translates to BP( is not trivial. This is because in RL the biases and variances are computed on more linear relationship between states and values, whereas in our case gradients have complex dependencies induced by the RNN Jacobian (cf. Eq. 15 Indeed, when we tried to directly analyse the bias and variance of the gradient vector, we were unable to obtain meaningful results. Instead in our results we simply quantify the goodness-of-fit by the cosine similarity metric and the overall RNN performance. In this case a high is consistently the best choice, but whether mid-range values can be beneficial for certain task conditions, as can happen in RL, remains to be tested in future work.
Additionally, whilst the complexity of BP()) does not depend on time, the algorithm is still expensive, particularly if there are many hidden units in the RNN (Ta For example, if there are [h] RNN units and a linear synthesiser is applied then, due to the requirements of storing and updating the synthesiser eligibility traces, the memory and computational complexity of the algorithm is O(h(3) and O(hb[) respectivcly (cf. Table 3): this is as costly as the notoriously expensive real-time recurrent learning algorithm (Williams & Zipser 1989). We postulate that one way to alleviate this issue is to not learn error gradients with respect to RNN activity directly; but instead some low dimensional representation of that error gradient. In particular, it has been recently demonstrated that gradient descent often takes place within small subspace Gur- Ari et al. (2018). In principle, therefore, the synthesiser could learn some encoding of the error gradient of dimensionality s << [h], significantly reducing the complexity of BP))). We predict that such low dimensional representations, which can reduce the noise of high dimensional error gradients, may also lead to more stable syuthesiser learning. Speculating further, it may be that this bottleneck role is played by the thalamus in the communication of predicted error feodback in cerchellar-cortico loops (sce previous scction; Pemberton al. (2021)). ).
Finally, we highlight that BP(A) requires the dynamics of the main model to be represented by sequentially dependent states; i.c. the Jacobian Shet1 # 0, as is inherently satisfied by RNNs. However, we note that ah. BP(A) may in principle be applied to feedforward networks where activities are now defined across layers
12

---

Published in Transactions on Machine Learning Research (04/2024)
instead of timesteps similar to what was done by Jaderberg et al. (2017) However, according to formulation (A) the same synthesiser parameters should operate across layers, thus future work should explore how to best relax this constraint.
7
Conclusion
BPTT can be expensive and enforce long waiting times before gradients become available. Synthetic gradients remove these locking constraints imposed by BPTT as well as the associated computational and memory complexity with respect to the task length (Jaderberg et al. 2017). However, the bootstrapped n-step algorithm for learning synthotic gradients as proposed by Jaderberg et al. can lead to biased estimates and also mainlains some dependence on (truncated) BPTT to be performed.
Inspired by the (D(A) algorithm in RL we propose novel algorithm for learning synthetic gradients: (A).). This algorithm applies forward propagating cligibility traces in order to reduce the bias of its estimates and is fully online. We thus extend the work of Jaderberg et al.) in developing computational bridge wetw estimating expected return in the RL paradigm and future error gradients in supervised learning.
Through a combination of analytical and empirical work we show that BP(A) outperforms the original implementation for synthetic gradients. Moreover, our model offers an efficient online solution for temporal supervised learning that is of relevance for both artificial and biological networks.
Acknowledgments
We would like to thank the Neural & Machine Learning group and the Joao Sacramento group for useful feedback. J.P. was funded by a EPSRC Doctoral Training Partnership award (EP/R513179/1 and R.P.C. by the Medical Research Council (MR/X00 (00/60006177),, BBSRC (BB/X013340/1) and a ERC-UKRI Frontier Research Guarantee Grant (EP/ HOP/Y27881//).. This work made use of the HPC system Blue Pebble at the University of Bristol, UK. We would like to thank Dr Stewart for donation that supported the purchase of GPU nodes embedded in the Blue Pebble HPC system.
References
Riccardo Avvisati, Anna-Kristin Kaufmann, Callum J Young, Gabriclla E Portlock, Sophie Cancemi, Rui Ponte Costa, Peter J Magill, and Paul D Dodson. Distributional coding of associative learning within projection-defnndd populations of midbrain dopamine neurons. bioRzin, 2022.
Kevin T Beier, Elizabeth E Steinberg, Katherine E DeLoach, Stanley Xie, Kazunari Miyamichi, Lindsay Schwarz, Xiaojing Gao, Eric Kremer, Robert Malenka, and Liqun Luo. Circuit architecture of vta dopamine neurons revealed by systematic input-output mapping. Cell, 162(3):622-66) -344, 2015.
Guillaume Bellec, Franz Scherr, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. Biolog- ically inspired alternatives to backpropagation through time for learning in recurret neural nets. arXiv preprint Xix::11911. 090099 2019.
Ellen Boven, Joseph Pemberton, Paul Chadderton, Richard Apps, and Rui Ponte Costa. Cerebro-cerebellar networks facilitate learning through feedback decoupling. Nature Communications, 14(1):1-18, 2023.
Wojciech Marian Czarnecki, Grzegorz Swirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukeuoglu. Understanding synthetic gradients and decoupled neural interfaces. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 904-912. JMLR. org, 2017.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1911)1141-142, 2012.
Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Corncil, and Johanni Brea. Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules. Frontiers in neural circuits, 12:53, 2018.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:44105401,, 2014.
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arviv preprint 1rX::::112.04744 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 988):1735-1788 1780, 1997. doi: 10.1162/neco.1997.3 199.....99......33... URL htttps://doi.org// 11t.///e....///....o
1735
Jeffrey Hollerman and Wolfram Schultz. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature neuroscience, 1(4):304-309, 1998.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using syuthetic gradients. In Proceedings of the 34th International Conference on Machine Learning-Voon Volume 70, pp. 1627 1635. JMLR. org, 2017.
Mitsuo Kawato, Shinya Kuroda, and Nicolas Schweighofer. Cerebellar supervised learning revisited: bio- physical modeling and degrocs-of-freedue of-rreeomm control. Current opinion in neurobiology, 21(5):791-800 2011.
Mitsuo Kawato, Shogo Ohmac, Huu Hoang, and Terry Sanger. 50 years since the Marr, Ito, and Albus models of the cerebellum. Neuroscience, 462:151 174, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint arXiv:1412 12. 6980, 2014.
Quoe V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXi::104..0000 2015.
Talia N Lerner, Carrie Shilyansky, Thomas J Davidson, Kathryn E Evans, Kevin T Beier, Kelly A Zalocusky, Ailey K Crow, Robert Malenka, Liqun Luo, Raju Tomer, et al. intact-brain analyses reveal distinct information carried by sne dopamine subcircuits. Cell, 6335-)6635-647 2015.
Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82 89, 2019.
Owen Marschall, Kyunghyun Cho, and Cristina Savin. Evaluating biological plausibility of learning algo-
rithms the lazy way. In Real Newrons Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence @ NeurIPS 2019, 2019. URL URL https::/ http://pparroiieeoooooooooooeeea wwt.://orrrurte---ggggtttt
Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework online learning algorithms for training recurrent neural networks. Journal of Machine Learning Research, 21(135):1 -3,, 2020.
Vijay Mohan K Namboodiri and Garret D Stuber. The learning of prospective and retrospective cognitive maps within neural circuits. Newron, (09(2))335523522) 2021.
Medina JF Ohmnae S. Plasticity of ponto-cerebellar circuits generates prospective error signal in climbing fiber. Program No. 579.01. Newroscience 2019 Abstracts. Chicago, IL: Society for Neuroscience, 2019. Online, 2019.
Joseph Pemberton, Ellen Boven, Richard Apps, and Rui Ponte Costa. Cortico-cerebellar networks decou- pling neural interfaces. Advances in Neural Information Processing Systems, 34, 2021.
Luke Y Prince, Roy Henha Eyono, Ellen Boven, Arna Ghosh, Joe Pemberton, Franz Scherr, Claudia Clopath, Rui Ponte Costa, Wolfgang Maass, Blake A Richards, et al. Current state and future directions for learning in biological recurrent neural networks: A perspective piece. arXiv preprint 2..iv:2105.000 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Lin Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint ar 2f::2011100000 2020.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Harm Van Seijen, A Rupam Mahmood, Patrick M Pilarski, Marlos C Machado, and Richard S Sutton. True online temporal-differnnce learning. The Journal of Machine Learning Research, 17(1):5057-5096 2016.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Newral compulation, 122)2770 280, 1989.
Sho Yagishita, Akiko Hayashi- Takagi, Graham C R Ellis-Davies, Hidetoshi Urakubo, Shin Ishii, and Haruo Kasai. A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science, 16116104),011--162 2014.
15

---

Published in Transactions on Machine Learning Research (04/2024)
A Proof of Theorem 31
As this section is more involved, for clarity denote scalar variables in italics and vectors in bold.
Approach: We demonstrate that Theorem 3.1 holds for arbitrary syuthesiser parameter @i (at the ith column and jth row of 0) that is,
02 - 0f5.A
0, as a 0.
Bfr 000
This is proved in Proposition A.7
For reference, we explicitly write the weight updates for paramcter @ according to the accumulate (P algorithm:
dh: ji
0g(h;; 0.)
e/' 7d Ōh,-
et-1
(18)
00.7
5 0L++1 t th
g(h;; 0.),
(19)
th,
oh,
0f 0f' + ad, et
(20)
also define the helper variable d,,
θh+ Shit1 6., ah.
Uh.
Oh.
+
g(h: 0,)
6,
(21)
Th,
Oh
oh.
In general, we follow the structure of the analogous proof for the accumulate TD(A) algorithm al. 2016
(an Soijen
A.1 Statements and proofs
Lemma A.1. G21t (++++1 G-llt (27) where
0L++1 Sh,
0h+t
Oh,
8/,
rg (g(h141500)
g(hr; 0.-1)
(22)
Th
th.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Proof:
G - Gaa =(1 X)
xm1 G(()) + +x--GG(t--
Definition: Equation 12
n=1
(1 A)
X G X 1 1
=(1 rit
G ((t-a) + X _ "G ( x at ((t
a
=x (++1aa) X i-a (((aa)
=\
G (((+1-a) (!(-a)
=\
1+1-a Σ
x-1
TLatk + +g
Ohit1
b-a
Eg-- "0-0g(hhs 0
Platk
Oh:
ah.
aha
Oha
Oho
--o 1l1+1 + 7 1t1- g(hr 0.)
Oh1+1
Oh:
xx
- ((11::0-11) 02-1)
aha
Tha 0hu+1
ah.
[0L+11 + (hi 01)
Th.
=(>))
g(hi; 0(h1;0,-1)
Tha
Uha
Tha
62,1
Lemma A.2. G.ll Ga Ma+t + Eb=e41 (71)
n.b
Proof: We apply Lemma
recursively:
G G + (dy) a,t-1
2 + (27)--2- 2-0 8' 80.-2) + (X7))-1-a8) -0, 2.1-1
GAlat1 + (A7))''' 82 .... + (Ay)* (1 11-1-1185 n---1
t-1
GAle+1 + Σ
(3:))-08'
b=at1
Lemma A.3. Gdlat1 8' + g(haaia-1))
Proof:
Gala+1 =G¹
Ibat1 + (
0ha+1
oha
Th,
dLat1 *g(ho++100
That1
-ho
Th
- g(rr: (1h)):0-17 Oh.
g(hni 00-1) Oh.
tha
Oh
8'aa + 0a-1)
Lemma A.4. E1 (72)) 08'
Gill g(hnn 0a-1)
Proof:
[-1
G G3 +
Σ
(72)6-08' 8'
Lemia A.2
b=01
8' + glhai 0a-1)+ + Σ
(32)6-a8
Lemia
bma+1
1-1
= ha + Σ (12) 8
bma
17

---

Published in Transactions on Machine Learning Research (04/2024)
Lemma A.5. ef¹¹ = =o==o
dhe Og(h. Uh.
062
Proof:
dhs
Og(hs: 0o)
6/¹ 7d thi-1
ji
e-1
002'
Uhb
0g(ho-1;00-1)
0g(hs: 0%)
7> 0ho-1
7) Oh6-2
eb +
00
a0,
Oh,
0h
0g(ho::0 O
Oh
Th 0g(hi; 01)
= (T>A)²
+ (7:)6-1
0ha-1
dh
Tho 006²
Oha-1
Oh
00
Thg
1;-0011)
0g(h:: 0s)
Uh
006-1
065
Ohn 0g( 0o)
Sh. 0g(h0001)
= -
+ (7:20-1
Uhu aegi
0h1
a0
Oh. 0g(
dg 0gs; 0%)
(12)
Ohs-1
000-
00g
(72)6- 6-a Ths 0g (hai On)
Th 00.
Lemma A.6. Eb=d(r))0-0 S0.b G.a g(ha; 02-1) + O(a)
Proof:
Using 8,,1 8', - -0(hh;;))) 0,)
ah, + g(hr; 11-1) oh
we have
Th.
[-
1-1
Oh.
+ g(hn; 0.-1) Oh:
En2
8abb
(m2))-0 8' - 10(p:h 0.)
Oh.
th.
bma
bma
Th
(12))-o0 0.b
[())) (g(h:;; 0.) - g( 0(-1))
oh.
b=a
b=a
Σ 80b +(a))
b=a G.llt g(ha: +
Lemma A.4
Proposition A.7. Let Og be the initial weight veclor and let 05s be an arbilrary parameter of 0. 0. be the weight at time t computed by accumulate BP(X), and 0 be the weight at time computed by the online
X -SG algorithm. Furthermore, assume that Et-1 A + 0, where A := (G g(h;; (p))::0)) and G is the X-weighted synthelic gradient which uses On for all synthelic gradient estimations. Then, for all time steps t:
0g)h
38
0.4
0
08
- 0
0/
0/
2
18

---

Published in Transactions on Machine Learning Research (04/2024)
Proof:
The updates according to accumulate (P(A) (Equations 19 to 20 follow
0f"
8% +
6
Definition: Equation 18
b=0
dhs ig( 0u)
0
Σ (r))0-0
Lemma A.5
aha
00%
6=0
1-1
Shu og ha: 0n)
0%
(7))) 5,
Tha O0
b=0 a=
0g(ha: 0a)
8%
(7:2a
8u,
Definition: Equation 21
00e
6=0 a=0
Og(hai 0n)
n
11
0% a
To.b
Fn.b
00
b=a
b= a=
0= b=g
1-1
Og(ha: 0a) O00
0g +
Σ
Gall g(ha: Oa 0a-1) + O(ai)
Lemma A.6
a=0
8
+
1-1
G g(ha: + 0 O(a))
0g haa 0n)
00%i
Where the last line holds as a - O. On the other hand, the updates according to the online X-SG algorithm produce
G g(hoi 0w)) T ag(hag 0a)
0..54
00
+a a
Definition: Equation 13
002
a=0
0g(hn; 0n) 00%
0g + Σ
GAll g(hni Oo) + O(a))
Hence
035 01
( opf 06.5)
//
((a)
2
0/ 0g 1
[0 - 0g4
C+O(a)
where
Σ G g(hai (()=:0))
ag( 0n)
t-1 || X A 2|
C=
30
a=0
From the condition that Ei=1 s + we have C> Therefore
0f'
0::0
+ 0
as a - 0
0
B Recursive definition of G
B.2 we provide a recursive definition of the X-weighted synthetic gradient G as defined in Equation 11 Note that this is similar, but not the same, as the recursive definition defined in A.3 in the
In Proposition
19

---

Published in Transactions on Machine Learning Research (04/2024)
supplementary material of Jaderberg al (2017). In particular, this definition strictly considers future losses as in the context of synthesiser target, whilst that provided in Jaderberg et al. (017)) also considers the loss at the current timestep (Le).
We first require the following Lemma Lemma B.1. Por any sequence (nn),21)
xn In (1- - 1) Σ n
Tk-
(23)
Proof: By the sum rule,
(1 - A)
X nn
34 = (1 A)
xa
n=1 k=1
k= m=
Now,
80
k-
Σ 1-
Σ
>a
7=
n=1
n=1
00
k-2
Σ X"
X"
1 *-1
A
-
1 1-X
So
8 n
(1- 1) Σ - 24 = = Σ
Σ n-1
k=1
k=1 v=
-1- A)
24
k- Ib
km
En
n=1
Proposition B.2. We can define the A-weighted synthetic gradient (Eq. 11
incrementally. In particular,
ah+
- Shit1
Thut1 Th)+0
G² = Oh,
+ +X(G)
Oh, + + - N)g(h 0,)
(24)
oh.
20

---

Published in Transactions on Machine Learning Research (04/2024)
Proof: Let T S be the sequence length and define G.7) = 0 for T > T. Then we can write G as
Git1 (1 1) Σ
(m))
1+1
==1
aL + + ghictitn:: 0ctn)
Th
= (1 - A)
0h++1
8h
n=1
Ol++1+k
1h+41+
= (1 2)
+ +(1 (1-A) Σ
\" n"-
1n11a"0 g0 pctn)
Oh++1
Ohu+1
#=1
km
==1
8
1 g0-1 1--1a-11 n-1 14+1+n
+ -1-) 1 \n X1117" "0g("""(hh""i" ghhhggtthhn0000 0ctn) 8hi+1
Lemma B.1
Ohtt1
x" 2 2 Olttn
0hu-n
+ 111 Σ x" 2m-2gm 2-12-11 g(hetni 0ctn-1)
0ht+1
0h++1
n=2
n=2
So
G (1 A) Σ X G
Ölikk
Ohetn Oh.
(1 A)
7 g(heenn 0ctn-1) 11)
Th,
aLeck
Thita
(1 A)
+11 A) Σ x" 1m117" g(betni Oc+n-1)
Oh.
Oh.
k=1
n=1
JL-n
Thitn Th.
ym n-1
+ - A)
xa 7y g(heenn P 01tn-1)
Lemma B.1
dh.
==1
n=1
ahtt1 +
20-27n--2 ihtt 0h++1
Thit1 + +11 - N)gg(hi 0:) Oh.
Th.41
Oh:
th+
n=
Ohitn
shu+1
+ +xy
(1 -1-1)
xm 2m 1- g(uutni 0 Ow-n-1)
0h++1
n=2
1h+t1 + A7
na n-2 Ihuta
Ohita
Ohy-1
+ +11 A)
Σ
2
(aan-1g(hsi g(eetni 1c+n-1)
Oh,
0h
Oh,
Oh,
1=
n=2
Oh1t1
+ +11 (1- 11-))g(h++1:0))
dh,
0L++1 +
Ohi+1
Th++1
+ +7(1 N)g(h++1;0. 0.)
Zh,
Oh.
Oh
C Experimental Details
For the experiments which analysc the alignment of synthetic gradients to true gradients (Figures and we an RNN with a linear activation function. For the experiments for which train the RNN using synthetic gradients (Figures and 5). use LSTM units in our RNN architecture (Hochreiter & Schmidhuber 1997).
The model output for a given task is a (trained) linear readout of the RNN (output) stale. The synthesiser network performs a linear operation (with a bias term) on the RNN hidden state (concatenation cell state and output state for LSTM) produce an estimate its respective loss gradient. The synthetic gradient at the final timestep is defined as zero. As in Jaderberg et al. (2017). the synthesiser parameters initialised at zero.
21

---

Published in Transactions on Machine Learning Research (04/2024)
When truncated BPTT with truncation size n is used for task sequence length T, the sequence is divided such that the first truncation is of size R where R = T mod n and each following truncation is of size
For (accumulate) (P) we often find it empirically important to use a discount factor y < 1. Except in the experiments using fixed RNNs (Fig. 2 for which y = 1, for all conditions we set 7 = 0.9.
We often observe it necessary to scale down the synthetic gradient before it is received by the RNN. We find this particularly necessary when using synthetic gradients alongside truncated BPTT, and in this case generally find scaling factor 0.1 optimal as in Jaderberg et al. (2017). We find that (P()) is less sensitive to this condition and choose scaling factors depending on the task (see below).
Data is provided to the model in batches. Though it is in principle possible to update the model as soon as the synthetic gradient is provided (e.g. every timestep), we find that this can lead to unstable learning, particularly when explicit supervised signals are sparse (as in the sequential-MNIST task). For this reason we instcad accunulate gradients over timesteps and update the model at the end of the batch sequence. We use an ADAM optimiser for gradient descent. on the model parameters (Kingma & Ba 2014].
All experiments are run using the PyTorch library. Code used for the experiments can be found the Github page: https://githut cow wwm/nouraamm/ppppnaaa...a..
The toy task experiments used to analyse gradient aligmment were conducted with an Intel 17-8665U CPU, where each run with a particular seed took approximately one minute. The sequential MNIST task and copy-repeat tasks were conducted on NVIDIA GeForee RTX 2080 Ti GPUs. Each run in the sequential MNTST task took approximately 3 hours or less (depending on the model used); each run in the copy-repeat task took approximately 12 hours or less.
C.1 Toy task used to analyse gradient alignment
To analyse gradient alignment for fixed RNNs in the toy task we provide the model an input at timestep 1 and the model is trained to produce target 2-d coordinate the unit-circle at the final timestep T where T = 10. The input is a (fixed) randomly generated binary vector of dimension 10. The task error is defined as the mean-squared error between the model output and target coordinate.
We also consider plastic RNNs which are themselves updated at the same time as the synthesiser. To ensure that the RNN has to discover the temporal association between input and target, and not simply readout a fixed value, in this case we consider 3 input/target pairs, where the targets are spread equidistantly on the unit circle. For this case we are interested in the limits of temporal association that can be capture and consider increasing sequence lengths T: specifically, we consider T multiples of 10 up to 100. We deem sequence length solved by a model with a certain init (i.e. given seed) if the model is able to achieve less that 0.025 averaged over the last 20 of 250 training epochs - for that sequence length and all smaller sequence lengths. To improve readability, for this task the training curves are smoothed using a Srrtzkyy-Golay filter (with a filter window of length 25 and polynomial of order 3).
For this task we provide the model in batches of size 10, where epoch involves 100 batches. The number of RNN units is 30 and the initial learning rate for the synthesiser is set 1 x 10-4 and 1 10-3 for the fixed and plastic RNN cases respectively.
Finally, to clucidate whether, like the TD(A) algorithm (sce c.g. Figure 2 in Van Scijen et al. (2016) (P(A) can only operate for modest learning rates, we Lest the model with different step sizes a and compare the resulting performance against the online X-SG algorithm (Figure 3. Here we consider a plastic RNN trained using the respective synthetic gradients with the task sequence length T 10. Note that in this case synthesiser weights are updated at each timestep (Eq. 14] as opposed to only at the end of the sequence once all gradients are accumulated. For these experiments standard stochastic gradient descent. is used.
C.2 Sequential-MNIST task
For the sequential-NNIST task we present a row-by-row presentation of a given MNIST image to the model
(Dngg 2012
Te et al. 2015 That is, the task is divided into 28 timesteps where at the ith timestep the
22

---

Published in Transactions on Machine Learning Research (04/2024)
ith row of 28 pixels is provided to the model. At the end of the presentation, the model must classify which digit the input represents. The task loss is defined as the cross entropy between the model output and the target digit.
For this task we provide the model in batches of size 50 with an initial learning rate of 3 10"4. The number of hidden LSTM units For (P) the synthetic gradient is scaled by a factor of 0.1.
During training, the models with the lowest validation score over 50 epochs are selected to produce the final test error.
To verify that BP( can operate on non-trivial tasks with simple learning optimizers, we also perform weight updates for this task with stochastic gradient descent with momentum term Ç 0.9 (Fig. 9b. In this case the learning rates for each network is fixed at 0.01.
C.3 Copy-repeat task
For the copy-repeat task (Graves et al. 2014h the model receives a delimiter before an 8-dimensional binary pattern length N and then a repeat character R The model must then repeat the binary pattern times followed by stop character. The total sequence length therefore x (R+ +)+++ +33 For easier absorption R normalised by 10 when consumed by the model.
We follow the curriculum in Jaderberg et al. 2017) and alternatively increment N and R when batch average less than 0.15 bits is achieved.
For this task we provide the model in batches of size 100. For the RNN and readout parameters we use an initial learning rate of 1 x 10-3, whilst we find a smaller learning rate of 1 x 10-6 for the synthesiser parameters necessary for stable learning. The number of hidden LSTM units 100. For BP(A) we do not apply scaling to the synthetic gradient.
23

---

Published in Transactions on Machine Learning Research (04/2024)
D
Supplementary Figures and Tables
alignment true gradient
BPTT
(n=2)
BPTT SG (n=3)
alignment true gradient
alignment to true gradient
first timestep (T-9)
1.0
1.00
1.0
0.95
0.8
0.5
0.90
BPTT SG (n=2)
0.6
T- 3
BPTT + SG (n=3)
T
BPTT + SG (n=4)
0.4
0.80
BPTT SG (n=
(P())
0.0
50
100 0
50 epoch
100
T-1T-2T-3T TATT2TT-TTTA1TTTTTTTT7TTT11
n=1
n=2
n=3
n=4
n=5
epoch
timestep from end
(BP(O))
n-step
Figure 6: Learning synthetic gradients with the n-step synthetic gradient (Eq. 9) as in (Jader- berg et al. 2017) in the toy task. Inputs are provided at timestep 1 and the corresponding target is only available at the end of the task at time 7 = 10. (a) Alignment between synthetic gradients and true gradients for a fixed RNN model across different timesteps within the task, where the synthetic gradients are learned with BPTT truncation size n = (left) and n = (right). Alignment is defined using the cosine similarity metric. (b) The average alignment over the last 10% of epochs in a across all timesteps. (c) Aligument of synthetic gradients at the first timestep of the task for diffcrent n; BP(1) (dotted green) is shown for reference. Results show average (with SEM) over 5 different initial conditions.
24

---

Published in Transactions on Machine Learning Research (04/2024)
T = 10
T 20
T= 30
1.0
BPTT (n=2)
0.8
final error (MSE)
error (MSE)
BPTT (n-3)
BPTT (n-4)
0.5
(n=5)
0.4
0.0
0.0
100
200
200
10
20
40
epoch
1.00
BPTT
SG (n=2)
error (MSE)
0.75
BPTT
3)_
error (MSE)
0.4
4)=4)
0.50
BPTT SG (n=5)
0.2
0.25
100
200
0
100
200
0
100
200
10
20
30
40
50
epoch
0.6
BP(0) (Jaderberg al.)
final error (MSE)
BP(0.5)
0.4
BP(1)
0.2
10
20
30
40
50
60
70
80
90
100
Figure 7: RNN learning using n-step synthetic gradients in the toy task. (a) (Left) Learning curves of RNNs which are updated using n-step truncated BPTT over different sequence lengths T. (Right) Task error at end of training for increasing T. (b) Same as (a) but n-step syuthetic gradients are also applied. (c) Task error for RNNs updated using (A) end of training for increasing T. Results show average (with SEM) over 5 different initial conditions.
25

---

Published in Transactions on Machine Learning Research (04/2024)
BP(A)
1.0
error (MSE)
+ A=
0.25
0.5
+ A= 0.5
+ 0
+ A=1
0.0
0.0
0.2
0.4
0.6
0.8
online A-return
1.0 4
error (MSE)
0.5
0.0
0.0
0.2
a
Figure 8: Effect of step-size a on performance in toy task for (top) BP(A) and (bottom) the online X-SG algorithm. Task errors are averaged over the last 10 epochs training (out of 100). Results show average (with SEM) over 5 different initial conditions.
no BPTT models
BPTT models
100%
100%
73%
no BPTT
73%
(96)
Uaderberg
accuracy (9%)
BP(O.5)
BPTT (n=
BPTT SG (n=2)
(n-3)
(n=3)
BPTT (n=4)
BPTT SG (n-4)
25%
25%
(n=5)
(n=5) 5)
20
40
60
80
100
0
20
40
60
80
100
Figure BP(A) in sequential MNIST task with stochastic gradient descent. (a) Validation accuracy during training for (P()) models. (b) Validation accuracy during training for models which learn synthetic gradients (SG) with n-step truncated BPTT in original implementation (Jaderberg et al. 2017): final performance of B1) (as in (b): dotted green) is given for reference. Results show average (with SEM) over different initial conditions.
26


Reviews:
[{"review_id": "sm4UzmMxxL", "reviewer": "Reviewer_ReJk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nI find the proposed algorithm very useful and interesting as any improvement on the ability of recurrent models to train online and approach performance of BPTT is important. The experiments, while arguably simplistic by the modern standards, follow the methodology of Jaderberg et al and do show a viable improvement due to the use of BP($\\lambda$). \n\n## Weaknesses\n\n### Somewhat confusing notation\n\nThe paper could simply the notation and potentially eliminate some of the symbols ($v_t$ for example which doesn't seem to have its own significance). If it stays it needs a formal introduction in eqs 5 and 6 which should probably be coupled with $h_t$, otherwise if we assume $v_t$ comes from its own marginal distribution, I'm not sure the loss still makes sense. Sometimes the paper uses the partial derivative notation, sometimes nabla operator. The main text uses normal fonts for (could be-) vectors but these are written in bold in the supplementary material. All the different sub- and superscripts for $G$ require memorization while reading. This may confuse the reader.\n\nDiagrams on Picture 1 should ideally be all vectorized.\n\nI would also suggest to write all weight updates (such as eq 13) in the form of $\\delta \\theta_k^t = \\ldots$, this will unify the style between the in-text equations and Algorithm 1 and possibly also improve readability. \n\n### Some clarity issues\n\nI find this statement contradictory: \"for an appropriately small learningrate α the synthesiser learns the true BPTT gradient $G_t$\" below Theorem 3.1. As I understand, it won't because $h_t$ likely doesn't have enough information to predict the true BPTT gradient.\n\n### Plastic RNNs lack explanation \n\nIt would help if the exact plastic RNN model is referenced and ideally outlined at least briefly,  even if in the supplementary material. \nIf it's the model by Miconi et al, then there really needs to be a discussion of how synthetic gradients fit into this construction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper builds upon the synthetic gradients framework and presents a new algorithm for learning synthetic gradients based on the idea of eligibility traces from RL. The new algorithm called accumulate BP($\\lambda$) doesn't rely on BPTT and allows to control the bias-cost tradeoff and experimentally attains better results in terms of these two objectives.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Comparison with RTRL\n\nGiven that the computational requirement of Accumulate BP is similar to RTRL (Williams & Zipser, 1989), wouldn't it make sense to provide both theoretical and experimental comparison to this method?\n\n### Variance analysis\n\nAs I understand, one reason why TD($\\lambda$) is efficient is due to the ability of controlling bias-variance tradeoff in the presence of high-variance gradients. What is the situation with supervised learning? In your experiments, $\\lambda=1$ seems to perform the best but it in the RL case that's usually not the case. If all we need is $\\lambda=1$ then why bother with the RL origins of the method? I feel like I'm missing something here but it probably also suggests that this needs to discussed, especially since the paper draws parallels to RL. \n\nRelatedly, does it make sense to have a plot similar to first one in Figure 2 of Seijen & Sutton, 2014? That would visualize not only dependency on $\\lambda$ but also the learning rate. From that plot it is particularly clear that the optimal choice of $\\lambda$ is tied to the choice of the learning rate. \n\n### Notation and figures refinement\nSee my comments in the previous section", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Eb22VTWdLh", "reviewer": "Reviewer_upTs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The connection with reinforcement learning can bear numerous fruits. The proposed method is a clear example. Overall such an approach is both original and can inspire follow-ups.\n- The idea is rather well explained at the start, though the final proposal is still hard to follow.\n- The experiments illustrate clearly the potential of the approach.\n- Limitations are clearly delineated.\n\nWeaknesses:\n- Unfortunately the final proposal is hard to follow. Some parts appear to lack some explanations (see suggestions below).\n- A few more experiments could complete the exposition to give a full perspective on the idea (see suggestions below). \n- The relevance of the approach beyond RNNs is not clear yet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an alternative to back-propagation through time using synthetic gradients in an online manner. \nThe idea stems from the analogy between the original synthetic gradients approach and temporal difference algorithms developed in reinforcement learning. A particular feat of the method is to propose an online method that can update parameters of the synthetizer without the help of an intermediate backpropagation through time and that can provide more accurate estimates than the previously proposed approach. Several experiments illustrate the potential of the approach on synthetic tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Comments/questions/suggestions:\n- I don't fully understand the last term in Eq 11. A priori it should be $G_t^{(T-t)}$ and it is not clear form the definition of $G_t^{(n)}$ that this matches $G_t$. It seems that it is rather $\\partial L_T /\\partial h_t$. I have probably missed something. But this means that some explanations would be beneficial.\n- Line (13) is quite hard to follow: on one hand t would be the time step. On the other hand there is a index k, which, I suppose, is the iteration of the learner of $\\theta$. In particular I would have thought that we would have $G_t^{\\lambda|t}$ rather than $G_k^{\\lambda|t}$. Similarly for the indexes of $h_k$. Again I may have missed something but explanations would be highly beneficial on that part.\n- On that thread, it would be good to use an index k in line (7) instead of t which was use for time. (A priori the iterations fo the algorithms and the time steps of the RNN are two different things).\n- The intuition on the proposed algorithm 3.4 is missing. Or, to be exact, it relies on knowledge from RL. The paper would highly benefit from a stand-alone explanation of the motivation of the algorithm.\n- Having a table summarizing the computational and memory complexities of (i) the proposed approach, (ii) classical BPTT, (iii) BPTT with synthetic gradients previously proposed, would be beneficial for the paper. \n- In the learning part of the experiments, what algorithm was used? SGD? Adam? How the method compares when using different optimizers?\n- (Optional) In the experiments, it could be interesting to showcase the benefits in terms of memory of the proposed approach compared to BPTT. For example, if the architecture is such that the batch-size is ceiled to some number by BPTT, it could be interesting to see how the proposed algorithm can benefit from the relaxed memory requirements by using larger batch-sizes.\n\nMinor comments:\n- The legend in Table 1 is hard to follow. The authors may reorganize it in a more readable way. \n- End of page 2: typo \"...both empirically can it alleviate...\"\n- End of first paragraph of section 4.1: typo \"...to learn perfectly model the true BPTT...\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CYayuFyfIs", "reviewer": "Reviewer_3HWT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n+ The supplementary material in Jaderberg et al. (2017) mentioned this idea as it is reasonable and can help incorporate longer amounts of temporal information into the gradient estimates without needing to backprop through time for all of them.\n+ Table 1 comparing the concepts between RL and synthetic gradients is a nice analogy to have (minor comment: instead of describing the rows in text it also seems like there is space to describe them inline and may slightly improve the readability)\n+ Overall the paper is clear, well-written, and well-motivated. \n+ The toy experimental results in Fig 2/3 are convincing and clearly validates the claim of being able to better approximate the true gradient.\n\n# Weaknesses\n+ The only weakness I see is that the experimental results on MNIST/copy-repeat are not directly comparable to results in Jaderberg et al. (2017), and also does not include all of their experimental settings (e.g., Penn Treebank). For this reason, I borderline mark the paper as not validating the claim as I do not easily see how to compare the experimental results with previously published results on these datasets", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the idea of discounted synthetic gradients as an alternative to BPTT when training a recurrent model, which is summarized in Figure 1 and section 2.1/2.2/2.3. The idea of discounted synthetic gradients is mentioned in the supplementary material of Jaderberg et al. (2017) but not experimentally evaluated. The paper under submission proposes an algorithm for learning these discounted gradients and experimentally validates them for sequential MNIST and a copy-repeat task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am very open to discussing the weakness I raise on the experimental results not being directly comparable to previously published results", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper meets both criteria for acceptance. Also, it has already undergone a revision to incorporate reviewer feedback. The reviewers didn't request further changes after the rebuttal, so I don't see any reason for the paper to be revised again.
Decision: Accept (poster)