Equilibrium Propagation with Continual Weight Updates
Authors: Maxence Ernoult
Julie Grollier
Damien Querlioz
Yoshua Bengio
Benjamin Scellier
Venue:  (2020)
Keywords: Biologically Plausible Neural Networks
Equilibrium Propagation

Abstract: Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching those of Backpropagation Through Time (BPTT), but with a learning rule local in space.
Given an input x and associated target y, EP proceeds in two phases: in the first phase neurons evolve freely towards a first steady state; in the second phase output neurons are nudged towards y until they reach a second steady state.
However, in existing implementations of EP, the learning rule is not local in time:
the weight update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically.
This is a major impediment to the biological plausibility of EP and its efficient hardware implementation.
In this work, we propose a version of EP named Continual Equilibrium Propagation (C-EP) where neuron and synapse dynamics occur simultaneously throughout the second phase, so that the weight update becomes local in time. We prove theoretically that, provided the learning rates are sufficiently small, at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss given by BPTT (Theorem 1).
We demonstrate training with C-EP on MNIST and generalize C-EP to neural networks where neurons are connected by asymmetric connections. We show through experiments that the more the network updates follows the gradients of BPTT, the best it performs in terms of training. These results bring EP a step closer to biology while maintaining its intimate link with backpropagation.

Under review as a conference paper at ICLR 2020
EQUILIBRIUM PROPAGATION WITH CONTINUAL WEIGHT UPDATES
Anonymous authors
Paper under double-blind review
ABSTRACT
Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learn- ing and Neuroscience, by computing gradients closely matching those of Back- propagation Through Time (BPTT), but with a learning rule local in space. Given an input and associated target y. EP proceeds in two phases: in the first phase neurons evolve freely towards first steady state; the second phase output neu- rons are nudged towards until they reach a second steady state. However, in existing implementations of EP, the learning rule is not local in time: the weight update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically. In this work, propose version of EP named Continual Equilibrium Propagation (C-EP) where neuron and synapse dynamics occur simultaneously throughout the second phase, so that the weight update becomes local in time. Such learning rule local both in space and time opens the possibility of an extremely energy efficient hardware implementation of EP. We prove theoretically that, provided the learning rates are sufficiently small, at each time step of the second phase the dynamics neurons and synapses follow the gradients of the loss given by BPTT (Theorem 1). We demonstrate training with C-EP on MNIST and generalize C-EP to neural networks where neurons are connected by asymmetric connections. We show through experiments that the more the network updates follows the gradients BPTT, the best it performs terms of training. These results bring EP a step closer to biology by better complying with hardware constraints while maintaining its intimate link with backpropagation.
INTRODUCTION
motivation for deep learning that few simple principles may explain animal intelligence and allow us to build intelligent machines, and learning paradigms must be at the heart of such principles, creating a synergy between neuroscience and Artificial Intelligence (Al) research. In the deep learning approach to AI (LeCun et al. 2015), backpropagation thrives the most powerful algorithm for training artificial neural networks. Unfortunately, its implementation on conventional computer dedicated hardware consumes
energy than the brain by several orders of magnitude (Strubell et al.) 2019). One path towards reducing the gap between brains and machines in terms of power consumption is by investigating alternative learning paradigms relying on locally available information, which would allow radically different hardware implementations: such local learning rules could be used for the development of extremely energy efficient learning-capable hardware. Investigating such bioplausible learning schemes with real-world applicability is therefore of interest not only for neuroscience, but also for developing neuromorphic computing hardware that takes inspiration from information-encooding dynamics and topology of the brain to reach fast and energy efficient Al (Ambrogio et al., 2018) Romera et al., 22088) In these regards, Equilibrium Propagation (EP) is an alternative style of computation for estimating error gradients that presents significant advantages (Scellier and Bengio 2017).
EP belongs to the family of contrastive Hebbian learning (CHL) algorithms (Ackle et al. 1985 Movellan 1991 Hinton 2002) and therefore benefits from an important feature of these algorithms: neural dynamics and synaptic updates depend solely information that is locally available. As CHL algorithm, EP applies to convergent RNNs, i.e. RNNs that are fed by a static input and converge to steady state. Training such convergent RNN consists in adjusting the weights so that the

---

Under review as a conference paper at ICLR 2020
steady state corresponding to an input produces output values close to associated targets y. CHL algorithms proceed in two phases: in the first phase, neurons evolve freely without external influence and settle to (first) steady state; in the second phase, the values of output neurons influenced by the target y and the neurons settle to a second steady state. CHI. weight updates consist in a Hebhian rule strengthening the connections between co-activated neurons at the first steady state, and an anti-Hebbian rule with opposite effect at the second steady state. A difference between Equilibrium Propagation and standard CHL algorithms is that output neurons are not clamped in the second phase but elastically pulled towards the target y.
second key property of EP is that, unlike CHL and other related algorithms. it is intimately linked to backpropagation. It has been shown that synaptic updates in EP follow gradients of recurrent backpropagation (RBP) (Scellier and Bengio, 2019) and backpropagation through time (BPTT) (Emoult et al., 2019). This makes it especially attractive to bridge the gap between neural networks developed by neuroscientists, neuromorphic researchers and deep learning researchers.
Nevertheless, the bioplausibility of EP still undergoes two major limitations. First, although EP is local in space, it is non-local in time. all existing implementations of EP the weight update is performed after the dynamics of the second phase have converged, when the first steady state is no longer physically available. Thus the first steady state has to be artificially stored. Second, the network dynamics have to derive from primitive function, which is equivalent to the requirement of symmetric weights in the Hopfield model. These two requirements are biologically unrealistic and also hinder the development of efficient EP computing hardware.
In this work, we propose an alternative implementation of EP (called C-EP) which features temporal locality, by enabling synaptic dynamics occur throughout the second phase, simultaneously with neural dynamics. We then address the second issue by adapting C-EP to systems having asymmetric synaptic connections, taking inspiration from Scellier et al. (2018) we call this modified version C-VF.
More specifically, the contributions of the current paper are the following:
We introduce Continual Equilibrium Propagation (C-EP, 3.1 a new version of EP with continual weight updates: the weights of the network are adjusted continually in the second phase of training using local information in space and time. Neuron steady states do not need be stored after the first phase, in contrast with standard EP where global weight update is performed at the end of the second phase. Like standard EP, the C-EP algorithm applies to networks whose synaptic connections between neurons are assumed to be symmetric and tied.
We show mathematically that, provided that the changes in synaptic strengths are sufficiently slow (i.e. the learning rates are sufficiently small), at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss obtained with BPTT (Theorem [ and Fig. 2 Section 3.3) We call this property the Gradient Descending Dynamics (GDD) property, for consistency with the terminology used in Ernoult et al. (019).
We demonstrate training with C-EP on MNIST, with accuracy approaching the one obtained with standard EP (Section 4.2).
Finally, we adapt our C-EP algorithm to the more bio-realistic situation of a neural network with asymmetric connections between neurons. We call this modified version C-VF it is inspired by the Vector Field method proposed in [Selllerr et (2018). We demonstrate this approach on MNIST, and show numerically that the training performance is correlated with the satisfaction of Gradient Descending Dynamics (Section
For completeness, we also show how the Recurrent Backpropagation (RBP) algorithm Almeida (1987); Pinedal (1987) relates to C-EP, EP and BPTT. We illustrate the equivalence of these four algorithms simple analytical model (Fig. 3) and we develop their relationship in Appendix A
2 BACKGROUND: CONVERGENT RNNS AND EQUILIBRIUM PROPAGATION
Convergent RNNs With Static Input. We consider the supervised setting, where we want to predict target y given input The model is recurrent neural network (RNN) parametrized by
2

---

Under review as a conference paper at ICLR 2020
and evolving according the dynamics:
8 F (æ,, 0).
F is the transition function the system. Assuming convergence the dynamics before time step T, we have 8e where is the steady state of the network characterized by
8. = F (I, 0) 0)
(2)
The number of timesteps T is hyperparameter that we choose large enough so that sT = 8, for the current value of 0. The goal is to optimize the parameter 0 in order to minimize a loss:
L
(8.,y))
Algorithms that optimize the loss L for RNNs include Backpropagation Through Time (BPTT) and the Recurrent Backpropagation (RBP) algorithm of Almeida 1987); Pineda 1987), presented in Appendix
Equilibrium Propagation (EP). EP (Scellier and Bengio 2017) is learning algorithm that computes the gradient L in the particular case where the transition function derives from scalar function Φ, i.e. with of the form F(x,, 8, 0) 04 (x, 8, 0). The algorithm consists in two phases (see Alg. of Fig. D. During the first phase, the network follows a sequence of states steady state denoted 84. In the second phase, an extra term B
$1; 82, ..... and converges pertubs the dynamics of the neurons (where / > 0 is scalar hyperparameter):: starting from the steady state o s., the network follows second sequence of states $1 $2 83
8 s.. ... and converges to
steady state denoted s! Scellier and Bengio 2017) have shown that the gradient of the loss L can be estimated based on the two steady states s. and s.. Specifically, in the limit - 0,
1 04 00
04 00 (e, 80,
Oc
(x, 8
(4)
00
3 EQUILIBRIUM PROPAGATION WITH CONTINUAL WEIGHT UPDATES (C-EP)
This section presents the main theoretical contributions this paper. We introduce new algorithm to optimize -- (Eq. 3): new version of EP with continual parameter updates that we call C-EP. Unlike typical machine learning algorithms (such as BPTT, RBP and EP) in which the weight updates occur after all the other computations in the system are performed, our algorithm offers mechanism which the weights are updated continuously as the states of the neurons change.
3.1 FROM EP TO C-EP: AN INTUITION BEHIND CONTINUAL WEIGHT UPDATES
The key idea to understand how to go from EP to C-EP is that the gradient of EP appearing in Eq. reads the following telescopic sum:
8
2 00
xx 2
ae
(I, s.. 0)
=
I
00
(x,
O
00
x. s
(5)
tm
global parameter gradient in EP
parameter gradient time C-EP
In Eq. 5 we have used that o/ s. and 8, 5 as co. Here lies the very intuition of continual updates motivating this work; instead of keeping the weights fixed throughout the second phase and
updating them at the end of the second phase based the steady states and s as in EP (Alg. 1 of Fig. []. the idea of the C-EP algorithm is to update the weights at each time of the second phase between two consecutive states s." and s (Alg. of Fig. lb. One key difference in C-EP compared to EP though, is that, in the second phase, the weight update time step t influences the neural states at time step t + 1 in a nontrivial way, as illustrated in the computational graph of Fig. 2 In the next subsection define C-EP using notations that explicitly show this dependency.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 EP
Algorithm C-EP (with simplified notations)
Input: y, 8, B, n-
Input: y, 0, 33,
Output: 8.
Output:
1: 90 + 0
>> First Phase
1: + 0
> First Phase
2: repeat
3:
Sc++1 - ds
a
(x,, 82, 0)
3: S1+1 +
repeat
8 (x, St,
4: until 8 8
as
until St Sx
5: Store 8+
6: %
Second Phase
so +- 8x
> Second Phase
6: repeat
7: repeat
8:
$t11
8
ab
r, (r
++1
24 Us
(x, of. 0)
3 Us
8:
Parameter Update at Time t 2 80
10:
9:
until
of
s.
Global Parameter Update
9:
0 t +
0 80
(44)
$5
11: 0 <- 0
04 06
8+ (s., 0)
10: until of and are converged.
Figure 1: Left. Pseudo-code of EP. This is the version of EP for discrete-time dynamics introduced in Ernoult et al. (2019). Right. Pseudo-code of C-EP with simplified notations (see section 3.2 for formal definition of C-EP). Difference between EP and C-EP. EP. one global parameter update is performed at the end of the second phase in C-EP, parameter updates are performed throughout the second phase. Eq. shows that the continual updates of C-EP add up to the global update of EP.
3.2 DESCRIPTION OF THE C-EP ALGORITHM
The first phase C-EP is the same as that of EP (see Fig. [ In the second phase of C-EP the parameter variable is regarded another dynamic variable 0, that evolves with time f along with 8t. The dynamics of & and 0g in the second phase of C-EP depend on the values of the two hyperparameters 3 (the hyperparameter of influence) and (the learning rate), therefore we write st and 0 to show explicitly this dependence. With now both the neurons and the synapses evolving in the second phase, the dynamic variables s/.0 and 0²-0 start from s
and 00 0 and follow:
Am s+1
0$
3.11
dl
B.
x, 8t
-
8t
Øs
Os
1A 0:
08.7 0,
3
n
0Φ
00
s+1
S
3.1
08.09
0
00
x. s.
8n
The difference in C-EP compared to EP is that the value of the parameter used to update 8t41 in Eq. (6) is the current 08-n, not 0. Provided the learning rate is small enough, i.e. the synapses are slow compared to the neurons, this effect is weak. Intuitively, in the limit - 0, the parameter changes are negligible so that 8.80 can be approximated by its initial value 08- A. Under this approximation, the dynamics of ...9 in C-EP and the dynamics of s in EP are the same. See Fig. for simple example, and Appendix A.3 for proof in the general case.
8.n
3.3 GRADIENT- DESCENDING DYNAMICS (GDD)
Now we prove that, provided the hyperparameter and the learning rate y7 are small enough, the dynamics of the neurons and the weights given by Eq. (6) follow the gradients of BPTT (Theorem [ and Fig.2) For a formal statement this property, we define the normalized (continual) updates of C-EP, as well as the gradients of the loss L = ll (8T,Y) after T time steps, computed with BPTT:
4

---

Under review as a conference paper at ICLR 2020
r =
r--1 =
Or =
8T-2
ST-1
ST
C
1st phase of C-EP
(2)
0 (1)
(O)
(0)
A
C--EP
A
ac 80
ac
ac
00
0
02
T
30T
ac
DC ÖsT
B.y
Ban
0sr-2
d
82
(BPTT((2 (2)
BBPTT (1)
BBPTT())
AC
A
(I)
AC-
BPTT
2nd phase of C-EP
Figure 2: Gradient-Den Dessending Dynamics (GDD, Theorem I.. In the second phase of Continual Equilibrium Prop (C-EP), the dynamics of neurons and synapses descend the gradients of BPTT, i.e. C
(t). The colors illustrate when corresponding computations are realized in C-EP and BPTT. Top left. st phase of C-EP with static input æ and target y. The final state 8y is the steady state 8+. Bottom left. Backprop through time (BPTT). Bottom right. 2nd phase of C-EP. The starting state 8.11 is the final state of the forward-time pass, i.e. the steady state s..
EP (t) = BPTT
CC-EP (3,n.t) 1 ( 9.1 8.9
BBPT (1) ōST HL
dC
3 $,
1
A C- EP (3,1,()
00999 0g.00
BBPTT(() (t) = 00r-i
More details about L and the gradients BPPTT (t) and BBPTT (t) are provided in Appendix that injecting Eq. (6) in Eq. (7), the normalized updates of C-EP consequently read:
Note
A (B, 3 ae
1 TT
(z.
In (r."0""000"")) o(_"))
ae
which corresponds to the parameter gradient at time t, defined informally in Eq. (5) The following result makes this statement more formal.
Theorem (GDD Property). Let 80, 81......,, ST be the convergent sequence of states and denote the steady state. Further assume that there exists some step K where 0 < K < T such that s, = ST = 8T-1 -T--. K. Then, in the limit 1) - O and 3 - 0, the first K normalized updates in the second phase of C-EP are equal to the negatives of the first K gradients of BPTT. i.e.
lim lim 8-0
C SEP CC-EP((3. (3,11 = BPPTT (I),
Vt 0, K:
70
lim
lim A EP (3, BPTT (1).
8
Theorem rewrites s41 8.n x s..0 - Bc BST_ and 01-7 00.0 7l Mor-. showing that in the second
ac
phase of C-EP, neurons and synapses descend the gradients of the loss L obtained with BPTT, with the hyperparameters 3 and playing the role of learning rates for sQ. and 0,.. respectively. Fig, illustrates Theorem with a simple dynamical system for which the normalized updates A C and the gradients BPTT analytically tractable see Appendix C for derivation details.
5

---

Under review as a conference paper at ICLR 2020
A (B,1,
1 2/ +
!
VerTT (t)
2
23
A (B,n,))
1 2
7
Dirraa (8) 24
88
2
2/3
7 0
Model: St+1 = (st + +0,, L' = 5
T + 00
0 (1 2/)* A. (8,1)
B 3-0 0
VREP (t)
2
(3,1) 2 23)
VREP (t)
2
Figure 3: Illustration of Theorem on simple model. The variables and 0 are scalars, the first
phase equation is $1+1 2 (sc + 8). the steady state is denoted 8. and the loss is L 2 For completeness, we also include the corresponding gradients of Recurrent Backpropagation (RBP) and the normalized updates of EP, denoted RRE and A EP respectively. The equivalence between C-EP, EP, RBP and BPTT holds in the general setting: see Appendix A for thorough study of their relationship.
NUMERICAL EXPERIMENTS
In this section, we validate our continual version of Equilibrium Propagation against training on the MNIST data set with two models. The first model is vanilla RNN with tied and symmetric weights: the dynamics of this model approximately derive from a primitive function, which allows training with C-EP. The second model is Discrete-Time Time RNN with untied and asymmetric weights, which is therefore closer to biology. We train this second model with a modified version of C-EP which we call C-VF (Continual Vector Field) as it is inspired from the algorithm with Vector-Field dynamics of Scellier et al.] (0188). Erroult et al. 2019) showed with simulations the intuitive result that, model is such that the normalized updates EP 'match' the gradients of BPTT (i.c. if they are approximately equal), then the model trained with EP performs as well as the model trained with BPTT. Along the same lines, show in this work that the more the EP normalized updates follow the gradients of BPTT before training. the best is the resulting training performance. We choose to implement C-EP and C-VF on vanilla RNNs to accelerate simulations (Ernoult et 2019).
4.1 MODELS: VANILLA RNNS WITH SYMMETRIC AND ASYMMETRIC WEIGHTS
Vanilla RNN with symmetric weights trained by C-EP. The first phase dynamics is defined as:
8t+1 = (W 8, + Wg x):
(10)
where is an activation function, W is symmetric weight matrix connecting the layers and W, is a matrix connecting the input æ to the layers s. Although the dynamics are not directly defined in terms of primitive function, note that $++1 Re 34
(st, W) with (ss, W) s WW s if we ignore the activation function a. Following Eq. (6) and Eq. we define the normalized updates this model as:
Ba
C (B, ( (s
C- C-EP (B, =
1 (SeAT
B.n -141
8.0
(1))
Note that this model applies to any topology as long as existing connections have symmetric values: this includes deep networks with any number of layers of the models used in the experiments. More explicitly, for a network whose layers of neurons are s0, st. ... N with W.. W connecting the layers sn+1 and 8 in both directions, the corresponding
Appendix [ detailed descriptions
primitive function is Φ (T*) W Wantt Sat1 + NT W x see Appendix E. E. for more details.

---

Under review as a conference paper at ICLR 2020
Vanilla RNN with asymmetrie weights trained by C-VE. In this model, the dynamics in the first phase is the same as Eq. (TO) but now the weight matrix W is no longer assumed to be symmetric, i.. the reciprocal connections between neurons are not constrained. In this setting the weight dynamics in the second phase is replaced by version for asymmetric weights: w
W + +3 3 (s
5, ) so that the normalized updates are equal to:
1 8.0 614 =.."))
1 B.n 3
A VF (B,
AW VF..7,).
(12)
Like the previous model, the vanilla RNN with asymmetric weights also applies to deep networks with any number of layers. Although in C-VF the dynamics of the weights is not one of the form of Eq. (6) that derives from a primitive function, the (bioplausible) normalized weight updates of Eq. (12)
can approximately follow the gradients of BPTT, provided that the values of reciprocal connections are not too dissimilar: this is illustrated in Fig. (as well as in Fig. T2 and Fig. 13 of Appendix E.6 and proved in Appendix [.2]
This property motivates the following training experiments.
100
Error (%)
T
K
Epochs
Test error rate (%)
C-VF-1h
80
Test
Train
f
C-VF-2h
EP-1h
2.00
0.13
(0.20)
30
10
30
60
EP-2h
1.95 + 0.10
(0.14)
100
20
50
40
C-EP-1h
2.28 + 0.16
(0.41)
40
15
100
C-EP-2h
2.44 0.14
(0.31)
100
20
150
20
C-VF-1h
2.43 + 0.08
(0.77)
40
15
100
C-VF-2h
2.97 0.19
(1.58)
100
20
150
50 100 150
Initial 4(e, 6o)(")
Figure Left: Training results on MNIST with EP, C-EP and C-VF. "#h" stands for the number of hidden layers. We indicate over 5 trials the mean and standard deviation for the test error (mean train error in parenthesis). (resp. K) is the number of iterations in the 1s (resp. 2nd) phase. For C-VF results, the initial angle between forward (0r) and backward (8,) weights is To (fr; 0b) = 0%. Right: Test error rate on MNIST achieved by C-VF as function of the initial v (0,, 6)).
4.2 C-EP TRAINING EXPERIMENTS
Experiments are performed with multi-layered vanilla RNNs (with symmetric weights) on MNIST. The table of Fig. 4. presents the results obtained with C-EP training benchmarked against standard EP training (Emoutt et al., 2019) see Appendix E for model details and Appendix[E]] F. training conditions. Although the test eтror of C-EP approaches that of EP. we observe degradation in accuracy. This is because although Theorem guarantees Gradient Descending Dynamics (GDD) in the limit of infinitely small learning rates, in practice we have to strike balance between having learning rate that is small enough to ensure this condition but not too small to observe convergence within reasonable number of epochs. As seen Fig. 5 (b), the finite learning rate 7 of continual updates leads to A C-EP (8, 17, curves splitting apart from the BPPTT( curves. As seen per Fig. (a), this effect is emphasized with the depth: before training, angles between the normalized updates of C-EP and the gradients of BPTT reach 50 degrees for two hidden layers. The deeper the network, the more difficult it is for the C-EP dynamics to follow the gradients provided by BPTT. As an evidence, we show in Appendix F2 that when we use extremely small learning rates throughout the second phase (6 < 0 + Mann Ag C-EP) and rescale up the resulting total weight update (0 0
Alrot + Vriny A0rot), recover standard EP results.
4.3 CONTINUAL VECTOR FIELD (C-VF) TRAINING EXPERIMENTS
Depending on whether the updates occur continuously during the second phase and the system obey general dynamics with untied forward and backward weights, can span large range of deviations from the ideal conditions of Theorem Fig. (b) qualitatively depicts these deviations with a model for which the normalized updates of EP match the gradients of BPTT (EP): with continual weight
7

---

Under review as a conference paper at ICLR 2020
updates, the normalized updates and gradients start splitting apart (C-EP), and even more if the weights are untied (C-VF).
Protocol. In order to create these deviations from Theorem andd study the consequences in terms of training, we proceed as follows. For each C-VF simulations, we tune the initial angle between forward weights (0t) and backward weights (0,) between and 180". We denote this angle (O;, 0,) see Appendix [..nnddxx..ooooo for the angle definition and the angle tuning technique employed. For each of these weight initialization, we compute the angle between the total normalized update provided by AC-VF 17, i) and the total gradient provided by BPTT, Le. /BPTT = Ek-1 BBPTT() on random mini-batcher before training. We denote this angle v (AC-VF (tot), - VBPTT in the discrete-time setting of Ernoult et al.] (2019)- see Appendix [E.4] for model details. We proceed in the same way for EP and C-EP simulations, computing V (A F. (tot), -VB.TT (tot)) and y (AC-EP (tot), - (tot) before training. We use the generic notation (toot) to denote the total normalized update. This procedure yields (xr, v) data points with = Vi (A(tot), - -BBPTT(to)))) (tot)) and y = test error, which are reported on Fig. (a) - see Appendix F.T for the full table of results.
C-VF, i.e. AC (B, 77, tor)
K-1
=t=
(tot).. (VPP) Finally for each weight initialization, we perform training
Results. Fig. 5 (a) shows the test error achieved on MNIST by EP, C-EP on the vanilla RNN with symmetric weights and C-VF on the vanilla RNN with asymmetric weights for different number of hidden layers as a function of the angle V (toot, <BPTT (tot)) before training. This graphical representation spreads the algorithms between EP which best satisfies the GDD property (leftmost point in green at ~ 20") to C-VF which satisfies the less the GDD property (rightmost points in red and orange at ~ 100%). As expected, high angles between gradients of C-VF and BPTT lead to high error rates that can reach 90% for V (A C-F (tot). -V 11 (tot)) over 100% More precisely, the inset of VF
Fig. Fig. shows the same data but focusing only on results generated by initial weight angles lying below 90", L.e. (0;, 0,) = {0', 22.5", 450, 67.5", 90% J. From standard EP with one hidden layer to C-VF with two hidden layers, the test error increases monotonically with V (A (toot, - VBPTT (tot)) but does not exceed 5.05% on average. This result confirms the importance of proper weight initialization when weights weight angle is of 08, the impact of untying the weights on classification accuracy remains constrained, as shown in table of Fig. 4.1 Upon untying the forward and backward weights, the test error increases by ~ 0.2% with one hidden layer and by ~ 0.5% with two hidden layers compared to standard C-EP.
untied, also discussed in other context (Lillicrap et al.) 2016). When the initial
DISCUSSION
Equilibrium Propagation is an algorithm that leverages the dynamical nature of neurons to compute weight gradients through the physics of the neural network. C-EP embraces simultaneous synapse and neuron dynamics, resolving the initial need of artificial memory units for storing the neuron values between different phases. The C-EP framework preserves the equivalence with Backpropagation Through Time: in the limit of sufficiently slow synaptic dynamics (i.c. small learning rates), the system satisfies Gradient Descending Dynamics (Theorem D. Our experimental results confirm this theorem. When training our vanilla RNN with symmetric weights with C-EP while ensuring convergence in 100 epochs, modest reduction in MNIST accuracy is seen with regards standard EP. This accuracy reduction can be eliminated by using smaller learning rates and rescaling up the total weight update at the end of the second phase (Appendix [.2) On top of extending the theory of Emoult et al. (019)) Theorem I also appears to provide a statistically robust tool for C-EP based learning. Our experimental results show as in Ernoult al. (2019) that, for a given network with specified neuron and synapse dynamics, the more the updates of Equilibrium Propagation follow the gradients provided by Backpropagation Through Time before training (in terms of angle this work), the better this network can learn.
Our C-EP and C-VF algorithms exhibit features reminiscent of biology. C-VF extends C-EP training to RNNs with asymmetric weights between neurons, as is the case in biology. Its learning rule, local in space and time, is furthermore closely acquainted to Spike Timing Dependent Plasticity (STDP), a learning rule widely studied in neuroscience, inferred in vitro and in vivo from neural recordings in the hippocampus (Dan and Poo| 2004). In STDP, the synaptic strength modulated by the relative timings of pre and post synaptic spikes within precise time window (Bi and Poo. 1998 2001).
8

---

Under review as a conference paper at ICLR 2020
100
Test error (%)
6-
+
EP-1h
80
EP-2h
4
C-EP-1h
C-EP-2h
40
C-VF-1h
20
25
50
75
C-VF-2h
0
20
40
60
80
100
120
140
160
180
Initial 4'A(tot),
(SPTT (tot)) (*)
Continual Updates
Untie the weights
EP
C-EP
1e-4
C-VF
1
0
-1
E3
-1
25
50
75
0
25
50
75
25
50
75
time step t n2nd phase)
time step (2 phase)
time step (2nd phase)
Figure 5: Three versions of EP: standard Equilibrium Propagation (EP), Continual Equilibrium Propagation (C-EP) and Continual Vector Field EP (C-VF). #-h denotes the number of hidden layers. (a): test error rate MNIST as function of the initial angle v between the total normalized update of EP and the total gradient of BPTT. (b): Dashed and continuous lines respectively represent the normalized updates (t (i.e. A (t), A C-PP (t), 40 CC-VF(() (t)) and the gradients - -V BPPTT (t). Each randomly selected synapse corresponds coincide for standard EP, they split apart upon untying the weights and using continual updates.
color. While dashed and continuous lines
Strikingly, the same rule that rate-based formulation, as shown through numerical experiments by Bengio et al. (2015). From this viewpoint our work brings EP step closer biology. However, C-EP and C-VF do not aim being models of biological learning per se, in that it would account for how the brain works or how animals learn, for which Reinforcement Learning might be more suited learning paradigm. The core motivation of this work is to propose a fully local implementation of EP, in particular to foster its hardware implementation.
for C-VF learning approximate STDP correlations in
When computed on standard computer, due to the use of small learning rates to mimic analog dynamics within a finite number of epochs, training our models with C-EP and C-VF entail long simulation times. With Titan RTX GPU, training fully connected architecture on MNIST takes hours 39 mins with hidden layer and 10 hours 49 mins with 2 hidden layers. On the other hand, C-EP and C-VF might be particularly efficient in terms of speed and energy consumption when operated on neuromorphic hardware that employs analog device physics (Ambrogio et al. 2018) Romera et al. 2018). To this purpose, our work can provide algorithm onto a neuromorphic system. Fig. 5 (a) shows that hyperparameters should be tuned so that before training, C-EP updates stay within 90" of the gradients provided by BPTT. More concretely in practice, it amounts to tune the degree of symmetry of the dynamics, for instance the angle between forward and backward weights see Fig. [.1] Our work is one step towards bridging Equilibrium Propagation with neuromorphic computing and thereby energy efficient implementations of gradient- gradient-based learning algorithms.
engineering guidance map
REFERENCES
D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. learning algorithm for boltzmann machines. Cognitive science, 911):147-16, 1985.
L. B. Almeida. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. volume 2, pages 609-618, San Diego 1987, 1987. IEEE, New York.

---

Under review as a conference paper at ICLR 2020
S. Ambrogio, P. Narayanan, H. Tsai, R. M. Shelby, I. Boybat, C. Nolfo, S. Sidler, M. Giordano, M. Bodini, N. C. Farinha, et al. Equivalent-aceuracl accelerated neural network training using analogue memory. Nature, 558(7708):00, 2018.
Y. Bengio, T. Mesnard, A. Fischer, S. Zhang, and Y. Wu. Stdp presynaptic activity times rate of change of postsynaptic activity. arXiv preprint wrxiv:1509.059883 2015.
G.-. Bi and M.-m. Poo. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. Journal of neuroscience, 18(24): 10464-10472, 1998.
G.-q. Bi and M.-m. Poo. Synaptic modification by correlated activity: Hebb's postulate revisited. Annual review of neuroscience, 24(1):139-166) 2001.
Y. Dan and M.-m. Poo. Spike 2004.
plasticity of neural circuits. Neuron, 44(1):23-30,
M. Ernoult, J. Grollier, D. Querlioz, Y. Bengio, and B. Scellier. Updates of equilibrium prop match gradients of backprop through time in an rnn with static input. arXiv preprint arXiv: 1905. 13633, 2019.
A. Hertz. Introduction to the theory of neural computation. CRC Press, 2018.
G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Y. LeCun, Y. Bengio, and Hinton. Deep learning. Nature, 521(7553):436, 2015.
P. Lillicrap, D. Cownden, D. B. Tweed, and C. J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7:13276, 2016.
J. R. Movellan. Contrastive hebbian learning in the continuous hopfield model. In Connectionist models, pages 10-17. Elsevier, 1991.
F. J. Pineda. Generalization of back pack-propaggttoon to recurrent neural networks. 59:2229-2232, 1987.
M. Romera, P. Talatchian, S. Tsunegi, F. A. Araujo, V. Cros, P. Bortolotti, J. Trastoy, K. Yakushiji, A. Fukushima, Kubota, et Vowel recognition with four coupled spin-torque nano-oscillators. Nature, 563(7730):230, 2018.
B. Scellier and Y. Bengio. Equilibrium propagation: Bridging the gap between energy-basce models and backpropagation. Frontiers in computational neuroscience, 11, 2017.
B. Scellier and Y. Bengio. Equivalence of equilibrium propagation and recurrent backpropagation. Neural computation, 31(2):312-329, 2019.
Scellier, Goyal, J. Binas, T. Mesnard, and Y. Bengio. Generalization of equilibrium propagation to vector field dynamics. arXiv preprint 1808 04873, 2018.
E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arxiv: 2019.
10

---

Under review as a conference paper at ICLR 2020
Appendix
A PROOF OF THEOREM
In this appendix, we prove Theorem D which we recall here.
Theorem (GDD Property). Let 80,81, $1...... 8T be the convergent sequence of states and denote
ST the steady state. Further assume that there exists some step K where 0 < K < I such that 8, ST = -T -T-K. K. Then, in the limit 1j - 0 and 3 + O, the first K normalized updates in the second phase of C-EP are equal to the negatives of the first K gradients of BPTT, i.e.
lim
lim 4 EP (3, t) = - -VBTT (t),
Vt 0, 0.I,.., K:
B
n=0
lim
lim
(3,n,t) = V BPTT (t).
8--0
A.1 A SPECTRUM OF FOUR COMPUTATIONALLY EQUIVALENT LEARNING ALGORITHMS
Proving Theorem I Jmmounts to prove the equivalence of C-EP and BPTT. In fact we can prove the equivalence of four algorithms, which all compute the gradient the loss:
1. Backpropagation Through Time (BPTT), presented in Section B.2
2. Recurrent Backpropagation (RBP), presented in Section B.3
3. Equilibrium Propagation (EP), presented in Section 2
4. Equilibrium Propagation with Continual Weight Updates (C-EP), introduced in Section 3
In this spectrum of algorithms, BPTT is the most practical algorithm to date from the point of view of machine learning, but also the less biologically realistic. In contrast, C-EP is the most realistic in terms of implementation in biological systems, while it is to date the least practical and least efficient for conventional machine learning (computations on standard Von-Neumann hardware are considerably slower due to repeated parameter updates, requiring memory access at each time-step of the second phase).
A.2 SKETCH OF THE PROOF
Theorem M can be proved in three phases, using the following three lemmas.
Lemma 2 (Equivalence of C-EP and EP). In the limit of small learning rate, iee. 0, the (normalized) updates of C-EP are equal to those of EP:
lim
A EP -
Vt 0:
7-
(n>0)
(13)
lim
A EP (3, (B,1))
7-
(n)00)
Lemma 3 (Equivalence of EP and RBP). Assume that the transition function derives from a prim- itive function, i.e. that F of the form F (x, s, 8)
24 8s (x, s, 8). Then, in the limit of small hyperparameter 3, the normalized updates of EP are equal to the gradients of RBP:
lim 6-0 (3>0) lim
EP (B,t) = PRBP (t),
Vt 02
(14)
AP (B, = -- RBP (t).
EP
6-+0
Lemma 4 (Equivalence of BPTT and RBP). In the setting with static input x. suppose that the network has reached the steudy state 8. after T --K steps, i.e. ST-K 8T- K+1 Then the first K. gradients of BPTT are equal the first K gradient of RBP, i.e.
8T = 8T 80.
BBTT (t) VIEr (t),
=t= 0, K:
BPTT (t)
v kBp
(t).
(15)
Proofs of the Lemmas can be found in the following places:

---

Under review as a conference paper at ICLR 2020
The link between BPTT and RBP (Lemma 2 is known since the late 1980s and can be found e.g. in Hertz (2018). We also prove it here in Appendix B
Lemma was proved in Scellier and Bengio (2099) in the setting of real-time dynamics.
Lemma is the new ingredient contributed here, and we prove it in Appendix A.3
Also a direct proof of the equivalence of EP and BPTT was derived in Ernoult et al. 2019
A.3
EQUIVALENCE OF C-EP AND EP
First, recall the dynamics of C-EP in the second phase: starting from son s. and 60¹0! 0 we have
s41
8.q
8.11
3
ae
(
Ōs
Os
Vt 0:
(16)
0.h 0g +
8,1
0g
8m .06..))
B
30
8411
00
2,
We have also defined the normalized updates of C-PP:
AC EP (3,n,t)=
1 3.
s..1)
3
14 0::
(17)
AC- C-FP EP (3,m.t)=
1
0..41 0
7
We also recall the dynamics of EP in the second phase:
Ol (s) 12
5 S+ and s111 Ôs
(2,5 0) -
(18)
Os
well the normalized updates of EP, defined in Ernoult al. 2019))
1
EP (3,1) = 641 - -s6
3
14 02:00
(19)
1 OΦ
O
(2, (,.0, 0))
AF (8,1)
(I, 14,841100
00
0e
Lemma 2 (Equivalence of C-EP and EP). In the limit of small learning rate, i.e. 0. the
(normalized) updates of C-EP are equal to those of EP:
lim
A C-EP (3.1,t) = A (B,t),
EP
Vt 0::
lim
A (3,1,t) = A
C-EP
(13)
(7200)
as We want to compute the limits of AC- (1, t) and (B, with v > O. First of all, note that under mild assumptions which we made here - of regularity on the functions & and l (e.g. continuous differentiabilitty. for fixed and f3, the quantities sf" and 0 are continuous as functions of 7) : this straightforward from the form of Eq. (16). As consequence, A C-EP (3, n, t) is a continuous function of n, which implies in particular that
Proof of Lemma 2
lim + (1)-))
A EP (3 A EP (8, 0,
(20)
Now, taking 1 0 in the bottom equation of Eq.
16) yields the recurrence relation 00.0 0,800 so
that 0..00 0..0 0 for every t. Injecting 00 in the top equation of Eq. (16) yields for s the
same recurrence relation as that of s (Eq. we have
18 so that st
8.0
syt for every t. Therefore, for n 0.
A CC-EP (3,0,t)
8.0
3.0
1 3
8 -sa) A (8,t).
(21)
t+1
3
follows from Eq.
20 and Eq.
that
lim
C
(8, n2, = A (8,
(22)
7 (7700)
12

---

Under review as a conference paper at ICLR 2020
Now let us compute lim (1>0)
A (3, Using Eq. (16).
have
A C-
E.
(3,
1
(+1 -f...)
(23)
1
1
aΦ
(3,, (x
am
0e.1)
(24)
3
00
Similarly before, for fixed t,
0+ 80
2. sf
0..1))
is continuous function of 17. Therefore
lim (120)
4 AC-EP (8,
1
I, s41
s411 08.00
8.00
" .0..0)
(25)
B
00
0e
1
I, 11t1;10) 0)
OΦ
I, 11
AB
EP (3, ()
(26)
B
00
30
consequence of Lemma is that the total update of C-EP matches the total update of EP in the limit of small 7), so that we retrieve the standard EP learning rule of Eq. (4) More explicitly, after K steps in the second phase and starting from 05'0 Oo:
K
0x
00
-0
1+1
7 EP (3,
(definition of Ag C-EE Eq.
1/
(AE (3,t)
(Lemma
t=
K
1
tΦ (x,, s641,00)
Σ
(I,, 8,41, 0o)
(definition of A Eq. 19
3
00
00
1=
7
OΦ
(vr,
0n)
O
(x, 8,, Op)
3
00
K
00
B EQUIVALENCE OF BPTT AND RBP
In this section, we recall Backprop Through Time (BPTT) and the Backprop (RBP) algorithm, which can both be used to optimize the loss of Eq. 3 Historically, BPTT and RBP were invented separately around the same time. RBP was introduced at time when convergent RNNs (such as the one studied in this paper) were popular. Nowadays, convergent RNNs are less popular: in the field of deep learning, RNNs are almost exclusively used for tasks that deal with sequential data and BPTT the algorithm of choice to train such RNNs. Here, we present RBP in way that it can be seen particular of BPTT.
Pineda Recurrent
B. SKETCH OF THE PROOF OF LEMMA
Lemma 4 which we recall here, is a consequence Proposition 5 and Definition 6 below..
Lemma 4 (Equivalence of BPTT and RBP). In the setting with static input suppose that the network has reached the steady state 8. after T steps, i.e. 8T-K
= 8T = 8T 8..
Then the first K. gradients of BPTT equal the first K gradient of RBP, Le..
BPTT (t) RBP (t), BPTT V RBP
Vt 0, 1, K
(15)
(t).
However, in general, the gradients BPTT (t) of BPTT and the gradients RBP (t) of RBP are not equal for > K. This is because BPTT and RBP compute the gradients of different loss functions:
13

---

Under review as a conference paper at ICLR 2020
BPTT computes the gradient of the loss after T time steps, i.. L (ST, y),
RBP computes the gradients of the loss at the steady state, i.e. L = e (8., V).
B.2 BACKPROPAGATION THROUGH TIME (BPTT)
Backpropagation Through Time (BPTT) is the standard method to train RNNs and can also be used train the kind of convergent RNNs that we study in this paper. To this end, we consider the cost of the state after T time steps, denoted L = l (ST: y), and we substitute the loss after T time steps L as proxy for the loss at the steady state L 8 (8.. v). The gradients of L can then be computed with BPTT.
To do this, we recall some of the inner working mechanisms of BPTT. Eq. ( rewrites in the form 81+1 F (z, St, 00+1 0). where O, denotes the parameter of the model time step the value being shared across all time steps. This way of rewriting Eq. ( enables us to define the partial derivative 80,
ac the sensitivity of the loss L with respect 0g when f1.... 0 0141....... Or remain
fixed (set to the value 0). With these notations, the gradient 09 reads as the sum:
0C = + + + 001
0C ac
OL
(27)
00 001
002
BPTT computes the 'full' gradient OC by first computing the partial derivatives ds, ae, iteratively, backward in time, using the chain rule of differentiation. In this work, we denote the gradients that BPTT computes:
Oc and ac
8e
0L
BBPTT(() (t) ŌST_
1t E 10. - 1]:
(28)
UL
BBPT((B (t) -0
Proposition 5 (Gradients of BPTT). The gradients v BPTT (t) and BBPTT (t) satisfy the recurrence relationship
dl
BPTT (0)
(ST y)
(29)
Ös
OF
1 1,2, ,,.,,
A BPTT (t)
(x, 8T T)T -BPTT 1),
(30)
8s
aF
Vt 1.2, .....
BPTT (t)
(x, 0)
BPTT 1).
(31)
00
B.3 FROM BACKPROP THROUGH TIME (BPTT) TO RECURRENT BACKPROP (RBP)
In general, apply BPTT, it is necessary store in memory the history of past hidden states 81, , in order to compute the gradients VBPTT(() (1) and VEPTT as in Eq. 30 31 However, our specific setting with static input x, if the network has reached the steady state s, after T K steps. i.c. if ST-K ST-K+1
ST-1 = = St, then we see that, in order to compute the first K gradients of BPTT, all one needs to know is 8F (z, sw, 0) and 8F 20 (I, S., 0). To this end, all one needs keep in memory is the steady state s.. In this particular setting, is necessary to store the past hidden states 8T, sr- ST-K since they are all equal to 8,,
De
The Mimeida-Pinedd algorithm (a.k.a. Recurrent Backpropagation, RBP for short), which was invented independently by Almeida (1987) and Pineda (1987), relies on this property to compute the gradients of the loss L using only the steady state s.. Similarly to BPTT, computes quantities v RBB (t) and VRBP (t), which we call 'gradients of RBP", iteratively for 0, 1,2,
14

---

Under review as a conference paper at ICLR 2020
Definition 6 (Gradients of RBP). The gradients V (t) and V (t) are defined and computed iteratively as follows:
0g
RBBP (0)
s y)
(32)
OF
14 2 0,
VRBP (t +
(x) 8,, 0)² -.RBP( (t),
(33)
as
aF
W 2 O,
B (t + 1) 00
(I, 3,, 0) V (t).
(34)
Unlike in BPTT where keeping the history of past hidden states is necessary to compute (or 'back- propagate") ) the gradients, in RBP Eq. 33 show that sufficient keep in memory the steady state only in order to iterate the computation of the gradients. RBP is more memory efficient than BPTT.
Algorithm 3 BPTT
Algorithm 4 RBP
Input: z. y, B.
Input: I, y. 0.
Outpul: 0.
Output: 0.
1: 80 4- 0
1: 8o 0
2: for = to T do
2: repeat
3:
8c+11 f- F (I, St, 0)
3:
S++1
F (3r, 81,
4: end for
4: until 8t = 8
5: VBPTT(() (0) 0 (ST: v)
88 V.BP (0) 0s
(s.. y)
6: for = 1 to T do
6: repeat
7:
(t))) t AF -T- T) Vt - 1)
7: VRBP (1) & 8F
(x, 8,, 0)
<RBP (1
88
8:
(l( t 00
aF
(x, st, 0) (1-1-1
8: RBBP (1) + oF
(x,, 8x, 0)
-RBP (t (t-1)
00
9:
end for
9:
until VRBP (1) = O.
10:
VBPTT (tot) + ETo (BPTT(()
10: VRBP (tot) < Ei RBB
Figure 6: Left. Pseudo-code of BPTT. The gradients (I) denote the gradients BBPTT (1) of BPTT. Right. Pseudo-code of RBP. Difference between BPTT and RBP. In BPTT, the state sy is required to compute aF (at, -r- f) and 30 (ær, -r. 0) thus is necessary to store in memory the sequence of states $1, $2,..... .., ST. In contrast, in RBP, only the steady state s, is required compute 0s
OF
BE (I,, Se, 0) and 29 (x, Sw, 0) it is not necessary to store the past states of the network.
B.4 WHAT 'GRADIENTS' ARE THE GRADIENTS OF RBP?
In this subsection we motivate the name of 'gradients' for the quantities RBB (t) and VRBP (t) by proving that they are the gradients of L* in the sense of Proposition 7 below. They are also the gradients of what we call the 'projected cost function" (Proposition using the terminology of Scellier and Bengio (2019).
Proposition 7 (RBP Optimizes L*). The total gradient computed by the RBP algorithm is the gradient of the loss L = l (8,, i.e.
OL
RBP
(35)
ae
RBBP(() (t) and VRBP(() can also be expressed as gradients of C, = É (st,y), the cost after t time steps. In the terminology Scellie and Bengio (2019), L, was named the projected cost. For = 0. 0. L, is the cost of the state projected a duration
Lo is simply the cost of the initial state 8g. For f
in the future.
Proposition 8 (Gradients of RBP are Gradients of the Projected Cost). The RBP gradients' RBBP (f)
and RBBP (t) can be expressed as gradients of the projected cost:
0L:
aL,
1A 2 0,
RBP (1)
RBP (1)
(36)
8s
00p
=
leg lg==
15

---

Under review as a conference paper at ICLR 2020
where the initial state Su is the steady state 8.-
Proof of Proposition 7
First of all, by Definition 6 (Eq. 32 34) is straightforward to see that
Vt - 0,
VRBP (t)
(OF
Ös (x, Sn, s0,0) 0)7)
Os (s..g): y)
0l
(37)
OF
( Os
[OF
(x, S0, 0)7)
de
14 2 1,
V
RBP (1)
(z, 80, 0)"
(8., 1)
(38)
00
as
Second, recall that the loss C is
L = e (su, y)
(39)
where
84 = F (Ir, 84, 0)
(40)
By the chain rule of differentiation, the gradient of L (Eq. 39p is
aL
de
Us.
(sx. v)
(41)
00
as
00
0s., we differentiate the steady state condition (Eq. 40) with respect to 0, which In order to compute 50 yields
ds. OF
ds.
OF
00 Os (a, 3. 0)
(z,, Sx, 0).
(42)
00
00
Rearranging the terms, and using the Taylor expansion (Id A) aF 0s (ar, S., 8), we get
Er A' with A
as. 00
OF Ōs
-1
OF
=
Id
(x, (c,s..0) s..0) 0)
00
(x. Sn, 0)
(43)
2
OF
2. OF
(x, S,, 0)
(I, Sw, 0)
(44)
Os
00
Therefore
OL
Ds.
(8+:1) y)
(45)
00
0s
20
2
dl
OF
(z, Sr, 0))
OF
(Sx, 8)
00 (I, 0)
(46)
Os
Os
V RBP
(4).
(47)
Proof of Proposition 8
By the chain rule of differentiation we have
0h++1
OF
T OL++1
(z, s0, 0)T Øs
(48)
Uso
Evaluation this expression for 80 = 8, we get
0c++
OF
T. Oc++1
(z. Sw, Q)T_ Os
(49)
asp
Is
180=8
18p=5.
Finally note that
0C
0C++1
dC,
(50)
Øs
Øs
3sp
80=8.
11
10=5
Therefore
ac, 8og
and RBP (t) satisfy the same recurrence relation, thus they are equal. Proving
the equality of 70
ac.
and VRBP(() (t) is analogous.
40 m8
16

---

Under review as a conference paper at ICLR 2020
C ILLUSTRATING THE EQUIVALENCE OF THE FOUR ALGORITHMS ON AN ANALYTICALLY TRACTABLE MODEL
Model. To illustrate the equivalence of the four algorithms (BPTT, RBP, EP and CEP), study simple model with scalar variable and scalar parameter 6:
1 (s, 0) 2
1 5= 2
50 = 0,
8141
L'
(51)
where 8, is the steady state of the dynamics (it is easy to that the solution is s, 8). The dynamics rewrites 81+1 = F (st, 0) with the transition function F (s, 0) = 2 (s + 0), and the loss rewrites L = (s.) with the cost function {(s) = 5 Furthermore, primitive function of the system is ((s, 0) = {s 0)2. This model has no practical application it is only meant for pedagogical purpose.
Backpropagation Through Time (BPTT). With BPTT, important point is that approximate the steady state s, by the state after T time steps sT, and we approximate L (the loss at the steady state) by the loss after T time steps (ST).
In order to compute (i.e. 'backpropagate") the gradients of BPTT, Proposition 5 Stells us that we need
compute (ST) ST,
OF (st, 0) and OF
(8e, 0) = 1.- We get
00
1,
BBPT (t)
8T
(8)
8T 2
(52)
2
Recurrent Backpropagation (RBP). Similarly, to compute the gradients of RBP, Definition de tells us that we need to compute 0s (sx) = Sx. Us
OF
(sx, 0) and
OF 80
(s., O) 3/4 We have
11 2 O,
RBP (1)
Sx
V
RBP
(1)
8
(53)
2
2
The state after T time steps in BPTT converges to the steady state 8. as T + therefore the gradients of BPTT converge to the gradients of RBP. Also notice that the steady state of the dynamics = 0.
Equilibrium Propagation (EP). Following the equations governing the second phase of EP (Fig. have:
A
1+1 =
St
8
+ 0.
(54)
2
2
This linear dynamical system can be solved analytically:
0
()) ))
YI 2 0,
8 1 23
1 + 23
(55)
Notice that of - A + 0: for small values of the hyperparameter 3, the trajectory in the second phase is close to the steady state s, = 6.
Using Eq. 19 it follows that the normalized updates of EP are
0
14 2 O,
A (3,0)) = 2 (1 (3-2
(F,(3 = 21+1 - (1-23)². (56)
Notice again that the normalized updates of EP converge to the gradients of RBP as 3 - 0.
Continual Equilibrium Propagation (C-EP).
The system of equations governing the system is:
1
{
$ 9.11 Sus
VI 0:
841
ann
2
B
st
B.n
2
(57)
0
0,
8-n (+1
00
+
1
8,+1
3.9
8,11
233
'The primitive function is determined up constant.
17

---

Under review as a conference paper at ICLR 2020
First, rearranging the terms in the second equation, get
(0.m 02.))
set1 -s ) 23
(58)
follows that
A C-EP (3,
2 -
C-EP (3,ntt)
(59)
Therefore, all need to do is to compute 4 over all indices from = 0 to - 1 we get
AÇ-EP(B, (B, t). Second, by iterating the second equation
0. - = 0 +
(s 8,0 s.
(60)
2
Using 8, 0 and plugging this into the first equation we get
s41
8.n
8t
8,
+
17
0.
(61)
2
43
4
Solving this linear dynamical system, and using the initial condition
s
9.9 = 0 we get
9,1
0
1 -
1] 23
+ 28
11
[
2B
(62)
1
32
23
2,3
20
Finally:
CC AC-EP(3,77t5
(3,
0
2,3
+
1)
(63)
2
23
18

---

Under review as a conference paper at ICLR 2020
D COMPLEMENT ON GRADIENT-DESCENDIA DYNAMICS (GDD)
Step-by-step equivalence of the dynamics of EP and gradient computation in BPTT was shown in Ernoult al. (2019) and was refered as the Gradient-Descenden this appendix, we first explain the connection between the GDD property of this work and the GDU property of Ernoult al. (2019). Then prove another version of the GDD property (Theorem) below), more general than Theorem
Updates (GDU) property. In
D.1
UPDATES (GDU) OF ERNOULT ET AL. (2019)
The GDU property of Ernoult et (0019) states that the (normalized) updates of EP are equal to the gradients of BPTT. Similarly, the Gradient- DesienninD Dynamics (GDD) property of this work states that the normalized updates of C-EP are equal to the gradients of BPTT. The difference between the GDU property and the GDD property is that the term 'update' has slightly different meanings in the contexts EP and C-EP. In C-EP, the 'updates' are the effective updates by which the neuron and synapses being dynamically updated throughout the second phase. In contrast EP, the 'updates' are effectively performed at the end of the second phase.
D.2 A GENERALISATION OF THE GDD PROPERTY
The Gradient Descending Dynamics property (GDD, Theorem states that, when the system dynamics derive from primitive function, i.e. when the transition function F is of the form F= 00 then the normalized updates of C-EP match the gradients provided by BPTT. Remarkably, even in the case of the C-VF dynamics that do not derive from primitive function 4 Fig. 5 shows that the biologically realistic update rule C-V follows well the gradients of BPTT. More illustrations of this property shown Fig. 12 Fig. [3] In this section we give theoretical justification for this fact by proving a more general result than Theorem
First, recall the dynamics of the C-VF model. In the first phase:
St+1 = a (W+s),
(64)
where a an activation function and W is a square weight matrix. In the second phase, starting from $b = Sy and W = W, the dynamics read:
Ol Is
844 = a (W...
.6
-
8.n
Vt N 0:
(65)
w w.a..
1 0.n
3.
3.1)
s
St.
B
Now let define the transition function F(s, W) o (W s), that the dynamics of the first phase rewrites
Sc+1 F (st, W).
(66)
As for the second phase, notice that aw (s, W) o' s) s, so that if we ignore the factor o' (W s), Eq. (65) rewrites
OF
de
3.11 = F
(t Bnn w?2n) -
Vt 2 0:
w w + 3 ow
OF
Bn w.S.))
Os
(5 5 4.11)
(67)
Now, recall the definition of the normalized updates of C-VF, as well as the gradients of the loss f. = (ST. y) after T time steps, computed with BPTT:
c C- VF (3,
3 ((4] sl
1
BPTT (t) = d
8sT
(68)
AC C--V VF (8,
(C-v((8,111)
(wan w!!")
V BPTT(()) WW
The loss L. and the gradients BPPTI (1) and BPTT (1) are defined formally in Appendix B.2
19

---

Under review as a conference paper at ICLR 2020
Theorem 9 (Generalisation of the GDD Property). Let So: S1.....87 8T be the convergent sequence of states and denote s, = ST the steady state. Further assume that there exists some step K where 0 < K S T such that 8, ST ST-1 ST. K- Finally, assume that the Jacobian of the transition function at the steady state symmetric, i.e. aF 8g (s., W) de (sx: W) Then, in the limit 1 + O and B + 0, the first K normalized updates of C-VF follow the the first K gradients of
aP
BPTT, i.e.
lim lim - 8-00 8-00
(3, ()t) -VBPTT(() (t),
Vt 0, 0.,,...,K K:
(69)
lim lim C-VF VF
[3,1,t) = --WWTT
W
7-0
few remarks need to be made:
1. Observe that
OF Os (s, o' o'( s)) -T
(70)
Ignoring the factor o' (W s). we see that if W is symmetric then the Jacobian of F is also symmetric, in which case the conditions of Theorem are met.
2. Theorem iis special case of Theorem 9 To see why, notice that if the transition function F is of the form F(s, W)
0 (s, W). then 0s
aF (s, W)
S2B (s. W)
aF (s, W
(71)
Ös
as
Ös
In this case the extra assumption in Theorem is automatically satisfied.
D.3 PROOF OF THEOREM 9
Theorem is a consequence of Proposition 5 (Appendix B.2). which we recall here, and Lemma 10 below.
Proposition 5 (Gradients of BPTT). The gradients BPPT (t) and DBPTT()) (t) satisfy the recurrence relationship
0l
BPTT
(ST, y)
(29)
Øs
OF
Vt = ..
BPTT (t) = (z, ST- 0) T VBPTT (t-1), 1),
(30)
Os
OF
Vt = 1,2 ,
V
BPTT (1)
(x, ST. T. (BPTT(t-1)
(31)
00
Lemma 10 (Updates of C-VF). Define the (normalized) neural and weight updates of C-VF in the limit 1j + and 3 + 0:
A C- (F) = lim lim A C-VF (3,,),)) (t)
Vt 0.1 K:
A CF bim lim A (8,11,t) 5-00-0
9-+0 700
(72)
They satisfy the recurrence relationship
dl
C- VF (0) = (8,,8),
(73)
Øs
OF
Vt 2 0,
A C- (t+ + 1+)
(I, 8,,0) 0) C- (t),
(74)
Øs
OF
Vt 2 0,
A C-VF (t+1)
(I, (I,5,,0) 0) -t)
(75)
00
The proof of Lemma 10 is similar to the one provided
Ernoult et al. 2019
20

---

Under review as a conference paper at ICLR 2020
E MODELS
In this section, describe the C-EP and C-VF algorithms when implemented multi-layered models, with tied weights and untied weights respectively. In the fully connected layered architecture model, the neurons no lateral connections within layer). We denote neurons of the n-th layer as s" with n E [0, N 1], where N is the number of hidden layers. Layers are labelled in backward fashion: the output layer, hidden layer (before the input layer). Thus, there are N hidden layers total. Fig. 7 shows this architecture with N = 2. Each model are presented here in a "real-time" and "discrete-time" settings For each model we lay out the equations of the neuron and synapse dynamics, we demonstrate the GDD property and we specify in which part of the main text they are used.
only connected between two consecutive layers (no skip-layer connections and
0 labels
the first hidden layer starting from the output layer, and N - the last
present this order:
1. Discrete-Time RNN with symmetric weights trained with EP,
2. Discrete-Timee RNN with symmetric weights trained with C-EP,
3. Real-Time RNN with symmetric weights trained with C-EP,
4. Discrete- Time RNN with asymmetric weights trained with C-VF,
5. Real-Time RNN with asymmetric weights trained with C-VF.
Our Discrete-Time RNN model' is also commonly called 'vanilla RNN' and it is refered to the "prototypical model' in Ernoult et al. (2019). Our (Real-Tima RNN model with symmetric weights' also commonly called "continuous Hopfield model' and is refered the "energy- model'
in Ernoult et al. (001)).
Demonstrating the Gradient Descending Dynamics (GDD) property (Theorem I on MNIST. For this experiment, we consider the 784-512-...5511 -51--10 network architecture, with 784 input neurons, 10 ouput neurons, and 512 neurons per hidden layer. The activation function used is (xx = (x). The experiment consists of the following: we take a random MNIST sample (of size 1 784) and its associated target (of size parameter e, we perform the first phase for 7 steps. Then, we perform on the one hand BPTT over K steps (to compute the gradients BPPTT), values of 3 and (to compute the normalized updates A C-EP or A C-VF) and compare the gradients and normalized updates provided by the two algorithms. Precise values of the hyperparameters T, K, 33 and 17 are given in Tab. E.6
10). For given value of the time-discretization
the other hand C-EP (or C-VF) over K steps for given
W21
W10
I
81
8
V
W2.
W 12
Wo
Figure Fully connected layered architecture with hidden layers.
E. DISCRETE-TIME RNN WITH SYMMETRIC WEIGHTS TRAINED WITH EP
Context of use.
This model is used for training experiments in Section 4.2 and Table 4.1
Equations with N 2. We consider the layered architecture of Fig. where o denotes the output layer, and the feedback connections are constrained to be the transpose of the feedforward connections, i.e. W WT W.--x- In the discrete-time setting of EP, the dynamics of the first phase
21

---

Under review as a conference paper at ICLR 2020
defined as:
[+ (Wor s
1A E [07
141
W12 si W 8.)
(76)
f+1
W2 + W 12
81)
In the second phase the dynamics reads:
0,3
1.8 Wol 8t
811
y
1A E [0, K]
1.8
2.8 W12 $
Wol 8t
0,3
(77)
(+1
W2 x + w12
1.8
141
As usual, y denotes the target. Consider the function:
① (I, s (6) = =s²)T Wor s² +s W12 + (s W2 æ.
(78)
We can compute, for example:
0$
W12 + W 8
(79)
TsT
Comparing Eq. (76) and Eq. (79.. and ignoring the activation function Ø, we can see that
s Os (x, (x. ? 11 5/-1, st-1)
(80)
And similarly for the layers 5 and s² According to the definition of A C-EP
in Eq. 19
for every layer and every E [0, K]:
A EP W2. (B, t)
$411
2.8
x
8t
2,8
A W1a (B,t) = 811
EP
1.3
2.3
1,8
8
2,8
(81)
EP
Wos (8,t) = - st1
0,3
6+1
1.3
8
Simplifying the equations with N 2. To go from our multi-layered architecture to the more general model presented in section 4.1. we define the state of the network as the concatenation of all the layers' states, i.. $ = (s², s! SOT and we define the weight matrices W and Wa as:
WT
W25 0
W
W 12
W 0
W
(82)
Wol
0
Note that Eq. 776 and Eq.
78 can be vectorized into:
8t+1 oW 5, W. x),
(83)
1 s W
S" W.
(84)
2
Generalizing the equations for any N. For general architecture with a given N, the dynamics of the first phase are defined as:
8+1 (Wor st)
Vt [0,
8 = (W Wint1 nnt WT W2-1n s (WN. N. & + W WN-IN' 87 x-11).
Vn [1,N 1]
(85)
8
and those of the second phase
22

---

Under review as a conference paper at ICLR 2020
0.3
Woi 86 1.8 B(y 50.0)
tt11
Vt E [0,K] K|
n.8
Won+1 8t 1+1.8 + W n- St s1
Vn e [1 -
(86)
N.8 a
WN x WN-IN 81
1,8
where y denotes the target. Defining:
N-1
((x, S. ...... (0)
Σ
s"
W nn-1 + S.
WN, 2,
(87)
ignoring the activation function o, Eq.
85
rewrites:
OΦ Onn
(2, N
80)
Vn E
(88)
According to the definition of 4
C-FP
in Eq.
19). for every layer W Wnn+1 and every t E [0, K]:
A EP WN., (3,1) = 1/2
(5+1 x
N.8
N.9 xx
(89)
EP
Wanti (3,t) = 3
1
8+1 8111 s
n.8 n41.99
8
n+1,8
E 10, 1]
Defining = (s ,
T
and:
0
WT N-IN
0
0
WN-
WN-IN
0
WT
0
0
W N.
N-2N-1
W
0
22-2N-1
W
(90)
0
0
W
0
0
Wo1
0
Eq. 85
and Eq.
87
can also be vectorized into:
8t Wa x)) 8141 oWW
(91)
1 3" W s
① 8, W. W,)
s" WT x.
(92)
2
Thereafter we introduce the other models this general case.
E.2 DISCRETE-TINE TIME RNN WITH SYMMETRIC WEIGHTS TRAINED WITH C-EP
Context of use.
This model used for training experiments in Section 4.2 and Table 4.1
Recall that we consider the layered architecture of Fig. 7 where S denotes the output layer. Just like in the discrete-time setting of EP, the dynamics of the first phase are defined as:
Equations.
0
++1 = σ (Wo1 ss))
1A E [0, T)
8 = σ W Wnn+1 S, n+1 + WT
n- In S; 1--1) 8t+1 = a (WN,r x. W WI-IN 8
Vn e [1,N- - 1]
(93)
N
N-1)
23

---

Under review as a conference paper at ICLR 2020
Again, in EP, the feedback connections constrained to be the transpose of the feedforward connections, i.c. W Wnn-1 = W In In the second phase the dynamics reads:
0,3.n
oWWi s1.8.1) B (y y 8: -0.9.1)
9+1
n.3.y
aWWn++ -8 111 W in 5t
Vn [1, 11,
Vt [)
S41
N.5.a
(WN,. + W N-IN sh sN-11.8m
641
0. + C n,
v0 E [Wnn+1}
(94)
As usual, y denotes the target. Since Eq. (93p and Eq.
85 are the same, the equations describing the
C-EP model also be written in vectorized block-wise fashion, in Eq. (91) and Eq. 92 We can consequently define the C-EP model in Section 4.1 per Eq. 10
According to the definitions of Eq. (6) and Eq.
7 for every layer Wnn+1 and every E [0,
WN.= (B,t) =
N.a 1/-
St
N.3.1 xx.T
(95)
C-EP A
Won+1 (3,t)
B
n.B.n 8141
8141
8t
84 2ut1,80"
Vn v [0,
E.3 REAL-TIME RNN WITH SYMMETRIC WEIGHTS TRAINED WITH C-EP
Context of use. This model has not been used in this work. We only introduce it for completeness
with respect to Ermoult al. 2019).
Equations.
For this model, the primitive function is defined
N Σ n=
N-1 +e Σ
(s s! , N-1) = (1-c)
||
((
2=0
(96)
so that the equations of motion read:
0
1+ = (1 e)s + eWo1 (ss))
Vt E [0, T)
s 41
n-1n -087 = (1 e) e(WN,. a (æ) WT W (s) N-11)
= [1 e)s + <(Wnn+1 a 1s + WT
Vn [L -
1+
In the second phase:
0,3.m
(1 - e)8t 0.3.1 - WWoi (s) (32..11) + Be(y 2..3.(11)
B.P.1
= (1 - 3.3. + (W (s) a1118.1 + WT n- (8, 1-)))))))))
Vn e [1, 1},
Vt E [0, K]
N.B.o
(1 e)8t N.8.1 + + N.z
(x) + WT WN-IN (s -.
0
0.5h +
C- E n,t) 4O ( (Wm++1)
(97)
where e iss a dineeetscrttintioonn parameter and y denotes the target.
According the definition of the C-EP dynamics (Eq. (6). the definition of A C-EEP explicit form of & (Eq. (E). for all time step E [0, KT. we have:
EP (Eq.
7 and the
C-EP Wx,x (B,1)()
1/
85t1
N.9.1)
(x)
St
N.9..)
a (x)
A
W W (3,7,
C-EP
t
8
n.8.1)
8141
1+1.8,m
(
( (2tt2,n)))
Vn E 10, N - 1]
E.4 DISCRETE- TIME RNN WITH ASYMMETRIC WEIGHTS TRAINED WITH C-VF
Context of use.
This model is used for training experiments in Section 4.3 and Table 4.1
24

---

Under review as a conference paper at ICLR 2020
Equations. Recall that consider the layered architecture of Fig. layer. The dynamics of the first phase in C-VF are defined as:
where s denotes the output
$4 oWoo sss)
Vt E [0:
$111
Wnn++ $t + W Win-1: 57
Vn [1
(98)
a (WN.= x WNN-1 -NN-I-S; 1)
S111
Here, note the difference with EP and C-EP: the feedforward and feedback connections are uncon- strained. In the second phase of C-VF:
0.8.1
(Wo1 1.8.7) ) + Bc(y 8t
D.B.1)
[41
n.P.n
(Wnn+1 2+1,8,11 Wnn-1 8t N,1,8.0
n 111
Vn E [1, N. 1},
St+1
AA E K]
$411
N.5.9
WN, WNN-1-88
0, + n 0 (B
40 (wnn+, Wn+1n}
(99)
As usual y denotes the target. Note that Eq. 98 can also be in a vectorized bisek-wise fashion as NN and provided that we define W and W, as:
Eq. 991 with = (s", s
WNN-1
0
0
WN-IN
0
WN-IN-2
0
0
W N
W =
0
-N-2N-1
0
0
W
=
(100)
0
0
0
W10
0
0
Woi
O
For all layers Win+1 and W Wn+in: and every [o, K) we define:
A C-VF (3, n..))
( N,.. N.B. )
WN.
C-VF
W Want1 (3,1)())
-8 mn. 86 +1,33
A C-VF
Witth (3,n,1)
141.9. 8t 211, n.8.1 =
E.5
REAL-TIME RNN WITH ASYMMETRIC WRIGHTS TRAINED WITH C-VF
Context of use.
This model used generate Fig. see Table E.6 for precise hyperparameters.
Equations.
For this model, the dynamics of the first phase are defined as:
++ = (1 e) + eWo (st)
0
VI E [0,
1+ - -) + (W a (sp + Wnn Wmn-1 no (8)-1)
Vn E N
211 - e)s +e (WN,z a (x) + WNN-1 (s,"-)
where e is the cime-iiscettzzatinn parameter. Again, as in the discre-time version of C-VF, the feedforward and feedback connections W W and W Wn-1n in unconstrained. In the second phase, the dynamics reads:
0.8.
(1 c) 0,3.m + eWoi . (s) 1.3,0
+ + (y- (y-sf.8. 6.8..))
a,0n (1 e)s² n... +e W (wnn11=
86 x1110m1) W Wnn-1 (s²-1.8m))
Vn [1, 11]
VI [0, K]
N.8.0 (1 c)s N.3.1 +e WN.,
(x) + WNN-1 (1-11
6-1
03.0 0:t0 + n VF (B, n.t) 4O (Win+1. W Wn+in}
C-
(101) where y denotes the target, as usual. For every feedforward connection matrix W Wxn+1 and every feedback connection matrix W Wntin. and for every time step E [0, K] in the second phase, we define
C-
(s St N.S.11 (xx)
C-VP (B, = 3
1
(5+11 - -st -5 =..8.1)) o (011,877) T
AC
C-VF (B,n,t)
1
(p+1 81 .1+11..1) .o -0(5 B))
T
W. W.l
B
25

---

Under review as a conference paper at ICLR 2020
E.6 FIGURES FOR THE GDD EXPERIMENTS
In the following figures, we show the effect of using continual updates with a finite learning rate in terms of the A C and -VBPTT processes different models introduced above. These figures have been realized either the discrete-time or continuous-time setting with the fully connected layered architecture with one hidden layer on MNIIT. Dashed an continuous lines respectively represent the normalized updates A and the gradients BBPTT. Each randomly selected synapse or neuron correspond to one color. We add an or synapse updates and gradients. Each C-VF simulation has been realized with an angle between forward and backward weights of degrees (i.e. (O,, 0b) 0%). For each figure, left panels demonstrate the GDD property with C-EP with 0 and the right panels show that, upon using ? > O, dashed and continuous lines start to split appart.
index to specify whether we analyse neuron
Table Table of hyperparameters used to demonstrate Theorem
Figure
Angle V (?)
Activation
T
K
Learning rates
C-EP C-VF
tanh
800
80
0,01
0.08
0
45
tanh
800
80
0.01
0.08
( -
C-EP
tanh
800
80
0.01
0.08
1.510
5 1.510 5
C-VF
45
tanh
800
80
0.01
0.08
1.510-5
1.510-5
C-VF
15
tanh
800
80
0.005
0.08
0
C-VF
14
15
0
tanh
800
80
0.005
0.08
2.10
2.10-3
C-VF
12
13
tanh
150
10
0.01
0
C-VF
13
tanh
150
10
0.01
2.10 5
2.10
C-EP
10
0
tanh
800
80
0.05
0.08
0 2.10
C-EP
TO
tanh
800
80
0.05
0.08
2.10-5
C-EP
tanh
150
10
0.01
0 - 0
C-EP
tanh
150
10
0.01
2.10
2.10
26

---

Under review as a conference paper at ICLR 2020
4
AC-EP EP
-VPPT
4 EP PPTT
1e-1 1
0
1
-1
2
4
8
0
2
6
4
EP
8PPT
4
EP
8
1e-1
51
1e-2
51
0.5
2.5
0.0
0.0
-0.5
-2.5
t
Figure Discrete-Time RNN with symmetric weights. Left: AÇ-EP (1) normalized updates
(1) 0) and -VBPTT (t) gradients. Right: AC-EP()) (t) normalized updates (1 0) and -BBTT (t) gradients.
4C EP
8PTT W.
EP
1e
le 2 W
Wa
1
0
8
2
t
1e-3
C
PPT
W
1e
AW,
BPTT
W.r
0
2
-2
0
2
6
8
2
6
t
Figure 9: Discrete- Time RNN with symmetric weights. Left: AC-EP(t
FP (t) normalized updates
(1) 0) and BBPTT(() gradients. Right: A (t)) normalized updates (n 0) and -BBPTT (t) gradients.
27

---

Under review as a conference paper at ICLR 2020
4
EP
VBPTT
4 EP BPTT
0
-1
-1
20
40
60
80
0
20
40
60
80
t
4 C- EP 8PPT
4
BPTT
1e-2
51
le-2
51
5
80
0
20
40
t
Figure 10: Real-Time RNN with symmetric weights. Left: AÇ-EP(t) normalized updates (7)
0) and BPPPTT(() gradients. Right: AC-EP(t) normalized updates (17 0) and BPTT gradients.
4C C- EP
VPPT
EP
-VPPTT
1e
le-3 3
W
W..
0
20
40
80
20
60
80
t
le
CC W..
PPT
W
1e-3
4W,
BPTT
0.5
0.0
0
20
40
60
80
20
40
60
80
t
Figure 11: Real-Time RNN with symmetric weights. L -P(())))))))))))) normalized updates (n 0) and
BtPTT(() gradients. Right: A (C-EP(t) (t) normalized updates (n > 0) and BBPTT (t) gradients.
28

---

Under review as a conference paper at ICLR 2020
---EP EP,
-PPPTT
- EP 8PTT
1e-1
2.5
0
0.0
0
4
6
0
6
4
BPTT
4
8PPT 5
1e-2
5
le-1
0.0
2.5
-0.5
0.0
6
Figure 12: Discrete-Time RNN with asymmetric weights. Left: VÇ-VF(() normalized updates
(1) 0) and -BBTTT (t) gradients. Right: -F normalized updates (1 0) and BBBTT (t)
gradients.
EP
BPTT
4 EP
BPTT Wa
le-2 2
Wn
1e
1
0.5
-0.5
4
6
8
0
2
6
t
EP
BPTT
4
EP
BPTT W
1e-3
W
W
1e 2
0.0
-2.5
0
4
6
8
0
6
t
C- EP
BPTT
C- EP
BPTT
1e-2 2
W., W
1e
W,
W
2
0
8
t
Figure 13: Discrete-Time RNN with asymmetric weights. Left: AC-VF (t) normalized updates
(1 0) and -VPPTT(() (t) gradients. Right: A (t)) normalized updates (n > 0) and -VPPTT (t) gradients.
29

---

Under review as a conference paper at ICLR 2020
--EEP EP
VBPTT
4 EP BPTT
1e-1
2.5
0
0.0
0
20
40
60
80
0
20
40
60
80
t
4C
8PTT
4
BPTT
le 2
5
1e-2
51
0
-2
-2
40
80
0
20
40
t
Figure 14: Real-Time RNN with asymmetric weights. Left: VC- normalized updates (7)
0) and BPPPTT(() gradients. Right: -F) normalized updates (1) 0) and BPPT gradients.
A EP
BPTT
EP.
BPTT
1e-3 3
1e 3
W
Wa
0.0
2
0
-2.5
20
40
60
80
0
20
40
60
80
t
A C- EP
BPTT
EP
BPTT
1e-4 4
Wio
W
1e 4
W
W1
0.5
0.5
0.0
-0.5
0.0
20
40
60
80
0
20
40
80
t
CC EP
BPTT
4
C
EP
PPT
1e-3
W.
W.
1e 4
W
1
0.0
-2.5
0
40
80
40
t
t
(t) Figure 15: Real-Time RNN with asymmetric weights. Left: AC-V((t) normalized updates (n
0) and -VPPTT(t) gradients. Right: (C-VF(t) normalized updates (1) > 0) and - BPPTT
gradients.
30

---

Under review as a conference paper at ICLR 2020
F EXPERIMENTAL DETAILS
F.1 TRAINING EXPERIMENTS (TABLE 4.1)
Simulation framework. Simulations have been carried out in Pytorch. The code has been attached to the supplementary materials upon submitting this work on OpenReview. We have also attached readme.tx txt with a specification of all dependencies, packages, descriptions of the python files as well the commands reproduce all the results presented in this paper.
Data set. Training experiments were carried out on the MNIST data set. Training set and test set include 60000 and 10000 samples respectively.
Optimization. Optimization was performed using stochastic gradient descent with size 20. For each simulation, weights were Glorot-initialized. No regularization technique was used and we did not use the persistent trick of caching and reusing converged states for each data sample between epochs as in Scellier and Bengio (017).
of
Activation function.
For training, we used the activation function
(x) + exp(-4(2 - 1/2))
(102)
Although is a shifted and rescaled sigmoid function, we shall refer to this activation function as 'sigmoid'.
Use of randomized B. The option 'Random S appearing in the detailed table of results (Table 3) refers to the following procedure. During training, instead of using the same 3 accross mini-batches only keep the same absolute value of B and sample its sign from Bernoulli distribution of probability z at each mini-batch iteration. This procedure was hinted at by Scellier and Bengio (2017) to improve test error, and is used in context improve the model convergence for Continual Equilibrium Propagation appearing as C-EP and C-VF in Table 4. training simulations.
Tuning the angle between forward and backward weights. In Table 4.1 initialized with different angles between the forward and backward weights denoted as V in 4.ble Denoting them respectively 0g and 0g the angle k. between them defined here as:
we investigate C-VF
Tr(O: 0 Tr(Or- TT) Tr(Ob 0..)
x(Or, 0p) = cos
where Tr denotes the trace, i.e. Tr(A)= As for any squared matrix A. To tune arbitrarily well enough n(fr. 0b), the procedure is the following: starting from 0, = 0y. i.c. x(0;, (.) 0, we can gradually increase the angle between Or and d by flipping the sign of an arbitrary proportion of components of 0g The more components have their sign flipped, the larger is the angle. More formally, we write 0, in the form A, (()) @ 0g and we define:
v (p) = x <0,, (p) ② 0z),
(103)
where (p) is mask of binary random values [+1, -1] of the same dimension of o5: (() -1 with probability p and (p) +1 with probability p. Taking the cosine and the expectation of Eq. (03), we obtain:
Tr(O; 0T)
Tr(O; 07] Tr(O: 0,
(cos(V (p))) =
+ (1 p)
Tr(0: 8T)
1 2p
31

---

Under review as a conference paper at ICLR 2020
Thus, the angle y between / and By @ (p) can be tuned by the choice of p through:
1 2
p VV)
(cos( 1))
(104)
Hyperparameter search for E.P. We distinguish between two kinds of hyperparameters: the recurrent hyperparameters i.e. T, K and / and the learning rates. A first guess of the recurrent hyperparameters I' and B is found by plotting the AC-EP and / processes associated to synapses and neurons to see qualitatively whether the theorem is approximately satisfied, and by conjointly computing
proportions of synapses whose AW A processes have the same sign as its BBPTT processes. K can also found out of the plots as the number of steps which are required for the gradients to converge. Morever, plotting these processes reveal that gradients are vanishing when going away from the output layer, i.c. they lose up to 10-1 in magnitude when going from layer to the previous (i.e. upstream) layer. We subsequently initialized the learning rates with increasing values going from the output layer to upstreams layers. The typical range of learning rates is [10- 10 J. [10, 1000] for T. [2, 100] for K and [0.01, 1] for B. Hyperparameters where adjusted until having train error the closest to zero. Finally, in order to obtain minimal recurrent hyperparameters i.e. smallest 7 and K possible we progressively decreased T and K until the
train error increases again.
Table 2: Table of hyperparameters used for training. "C" and "VF" respectively denote "continual" and "vector-ficld", #=# stands for the number of hidden layers. The sigmoid activation is defined by Eq. (102).
Activation
T
B
Random
Epochs
Learning rates
EP-1h
sigmoid
30
10
0.1
False
30
0.08 - 0.04
EP-2h
sigmoid
100
20
0.5
False
50
0.2 0.05 0.005
C-EP-1h
sigmoid
40
15
0.2
False
100
0.0056
0.0028
C-EP-1h
sigmoid
40
15
0.2
True
100
0.0056
0.0028
C-EP-2h
sigmoid
100
20
0.5
False
150
0.01
0.0018 0.00018
C-VF-1h
sigmoid
40
15
0.2
True
100
0.0076
0.0038
C-VF-2h
sigmoid
100
20
0.35
True
150
0.009 - 0.0016 0.00016
32

---

Under review as a conference paper at ICLR 2020
Table 3: Training results MNIST with EP, C-EP and C-VI. "#h" stands for the number of hidden layers. We indicate over five trials the mean and standard deviation for the test error, the mean error in parenthesis for the train error. T (resp. K) the number of iterations in the first (resp. second) phase.
Full table of results. Since Table 4.1 does not show C-VF simulation results for all initial weight angles, provide below the full table of results, including those which were used to plot Fig.
Initial (Or, 0b) (*)
Error (%)
T
K
Random 3 Epochs
Test
Train
EP-Ih
2.00 + 0.13
(0.20)
30
10
No
30
EP-2h
1.95 + 0.10
(0.14)
100
20
No
50
C-EP-Ih
2.85 0.18
(0.83)
40
15
No
100
C-EP-Ih
2.28 0.16
(0.41)
40
15
Yes
100
C-EP-2h
2.14
(0.31)
100
20
No
150
C-VF-1h
2.43 0.08
(0.77)
40
15
Yes
100
22.5
2.13
(0.74)
40
15
Yes
100
45
2.06
(0.78)
40
15
Yes
100
67.5
0.13
(0.81)
40
15
Yes
100
90
0.18
(0.78)
40
Yes
100
112.5
(2.92)
40
15
Yes
100
135
86.61
(88.51)
40
15
Yes
100
157.5
91.08 0.01
(90.98)
40
15
Yes
100
180
92.82 3.47
(92.71)
40
15
Yes
100
C-VF-2h
4.19 0.19
(1.58)
100
20
Yes
150
22.5
3:54
(2.70)
100
20
Yes
150
45
0.78
(2.86)
100
20
Yes
150
67.5
0.92
(4.68)
100
20
Yes
150
90
1.17
(4.81)
100
20
Yes
150
112.5
20.33 13.03
(20.30)
100
20
Yes
150
135
(60.53)
100
20
Yes
150
157.5
77.90 + 13.49
(78.04)
100
20
Yes
150
180
74.17 + 12.76
(74.05)
100
20
Yes
150
33

---

Under review as a conference paper at ICLR 2020
F.2 WHY C-EP DOES NOT PERFORM AS WELL AS STANDARD EP
We provide here further ground for the training performance degradation observed on the MNIST task when implementing C-EP compared standard EP. In practice, when training with C-EP, have to make a trade-off between:
having a learning rate that is small enough so that C-EP normalized updates are subsequently close enough to the gradients of BPTT (Theorem J
2. having a learning rate that is large enough to ensure convergence within a reasonable number of epochs.
In other words, the degradation of accuracy observed in the table of Fig. 4.1 is due using learning rate that is too large to observe convergence within 100 epochs. To demonstrate this, we implement Alg. which simply consists in using a very small learning rate throughout the second phase (denoted as ), ), and artificially rescaling the resulting weight update by bigger learning rate (denoted as 17). Applying Alg. 5 to fully connected layered architecture with one hidden layer, T 30, K 10, = 0.1, yields 2.06 + 0.13% test enror and 0.18 + 0.01% % train error over 5 trials, where we indicate mean and standard deviation. Similarly, applying Alg. too fully connected layered architecture with two hidden layers, T 100, K 20, 3 = 0.5, yields 1.89 + 0.22% test error and 0.02 + 0.02% train crror. These results are exactly the same as the one provided by standard EP see Table 3
Algorithm 5 Debugging procedure of C-EP
Input: y, 0, B3, 7) Menuy 10 10-bn.
Output:
1: 80 4- 0
First Phase
2: A0 <- 0
Temporary variable accumulating parameter updates
3: repeat
4:
5r+11 +
01
(xr, St: 0)
88
5: until =
6:
8
f- 8
D. Second Phase
7: repeat
8:
ot+1
00 38
I, 8 0
8
33
9:
4- 0 +
Tiins
8
1++1
36
11:
AO 20
Thiny
20
+++
20
12:
until sf and are converged.
13: <- A
7 A6
Rescale the total parameter update by
7stny
1stny
34

---

Under review as a conference paper at ICLR 2020
20
20
20
Error rate (96)
Error rate (96)
Error rate (96)
15
15
15
10
10
10
100
100
50
100
150
Epochs
Epochs
Epochs
Figure 16: Train and test eтror achieved on MNIST with Continual Equilibrium Propagation (C-EP) on the Discrete-Time RNN model with symmetric weights. Plain lines indicate mean, shaded zones delimiting mean plus/minus standard deviation over 5 trials. Left: C-EP on the fully connected layered architecture with one hidden layer (784-512-10) without beta randomization. Middle: C-EP the fully connected layered architecture with one hidden layer (784-512-10) with beta randomization. Right: C-EP on the fully connected layered architecture with two hidden layers (784-512-512-10) without beta randomization.
35

---

Under review as a conference paper at ICLR 2020
4(O, 06) = 0
4(O,, 06) 22.5
4(O,, 0b) 45
100
100
100
Test error (%)
Test error (%)
Test error (%)
75
75
75
50
50
50
25
25
0
0
50
100
0
50
100
50
100
Epochs
Epochs
Epochs
4(O, 0b) 67.5
v(O,, 0₂) 90
4(0, 0₂) 112.5
100
100
100
Test error (%)
Test error (%)
Test error (%)
75
75
75
50
50
50
25
25
0
0
100
0
50
100
50
100
Epochs
Epochs
Epochs
4(O, 0b) 135
v 0b) 157.5
V(O, 0,) 180
100
100
100
Test error (%)
Test error (%)
Test error (%)
75
50
50
50
25
25
25
50
100
50
100
50
100
Epochs
Epochs
Epochs
Figure 17:
Train and test error achieved on MNIST by Continual Vector Field Equilibrium Propagation (C-VF) on the Discrete- Time RNN model with asymmetric weights with one hidden layer (784-512. 10) for different initialization for the angle between forward and backward weights (V), Plain lines indicate mean, shaded zones delimiting
plus/minus standard deviation over 5 trials.
36

---

Under review as a conference paper at ICLR 2020
4(O,, 06) 0
4(O,, 0b) 22.5
4(O,, 0b) 45
100
100
100
Test error (%)
Test error (%)
Test error (%)
75
75
75
50
50
50
25
25
0
0
100
0
100
100
Epochs
Epochs
Epochs
4'O, 0b) 67.5
v'O,, 0₂) 90
4(O, 0b) 112.5
100
100
100
Test error (%)
Test error (%)
Test error (%)
75
75
75
50
50
50
25
25
25
0
0
100
100
100
Epochs
Epochs
Epochs
4e0, 0b) 135
v 0b) 157.5
4 0,) 180
100
100
100
Test error (%)
Test error (%)
Test error (%)
75
50
50
50
25
25
25
100 Epochs
100
100
Epochs
Epochs
Figure 18:
Train and test error achieved on MNIST by Continual Vector Field Equilibrium Propagation (C-VF) on the vanilla RNN model with asymmetric weights with two hidden layers (784-512-512-D1 for different initialization for the angle between forward and backward weights (V)). Plain lines indicate mean, shaded zones delimiting plus/minus standard deviation over 5 trials.
37


Reviews:
[{"review_id": "rygnS_8nYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is concerned with biologically plausible models of learning. It takes equilibrium propagation -- where updates depend on local spatial information, in the sense that the information is available at each neuron -- and modifies the algorithm so updates are also local in time, thus obtaining C-EP. The key insight is that the updates in EP can be written as a telescoping sum over time points, eq (5).\n\nMaking equilibrium propagation more biologically plausible is an interesting technical contribution. But, taking a step back, the setup is misguided. It is true that humans can solve classification problems. And various animals can be trained to do so as well. However, it should be obvious that animals learn to solve these problems by reinforcement learning -- they are literally given rewards like sugar water for correct answers. \n\nMNIST is an unusual dataset with a stark constrast between foreground and background that is far from biologically plausible. I know it has a long and important history in machine learning, but if you are interested in biologically plausible learning then it is simply the wrong dataset to start with from both an evolutionary and developmental perspective. It’s not the kind of problem evolution started with, nor is it the kind of problem human babies start with. \n\nMaybe C-EP can be repurposed into a component of some much larger, biologically plausible learning system that does a mixture of RL and unsupervised learning. Maybe not. The MNIST results provide no indication.\n\nThe authors have done a lot of solid work analysing BPTT, RBT, and C-EP. I suspect they are far more interested in understanding and designing efficient mechanisms for temporal credit assignment than they are in biological learning. That work can and should stand on its own feet.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1e010yl9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper introduces a new variant of equilibrium propagation algorithm that continually updates the weights making it unnecessary to save steady states. The also mathematically prove the GDD property and show the effectiveness of \n their algorithm (Continual-EP) on MNIST. They also show C-EP is conceptually closer to biological neurons than EP.\n\nThis paper tackles an important problem in bridging the gap between artificial neural networks and biological neurons. It is well-motivated and stands well in the literature as it improves its precedent algorithm (EP). The contributions are clear and well-supported by mathematical proofs. The experiments are accurately designed and results are convincing. I recommend accepting this paper as a plausible contribution to both fields. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxSJJoJ9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think it is an intriguing paper, but unfortunately left me a bit confused. I have to admit is not a topic I'm really versed in, so it might be that this affected my evaluation of the work. But also, as a paper submitted to ICLR I would expect the paper to be self-contained and be able to provide all the details needed. \n\nI do appreciate the authors providing in the appendix the proofs of the other theorems even if they come form other works. \n\n\nThe paper introduces C-EP, an extension of  a previously introduced algorithm EP, such that it becomes biologically plausible. In particular EP is local in space but not in time (you need the steady state of the recurrent state after the first stage at the end of the second stage to get your gradients). I think this is fair, and the need for biological plausibility is well motivated in the beginning of the work.  \n\nMy first issue is with the proof for the equivalence between EP and C-EP. This is done by taking the limit of eta goes to 0. I think I must be missing something. But the proof relies on eta being small enough such that \\theta_i = \\theta (i.e. theta does not change). Given this state evolves the same way as for EP, because we basically not changing theta. \nYet the crux of my issue is exactly here. The proof relies on the fact that we don't change theta. So then when you converged on the second phase, isn't theta the same as theta_0? So you haven't actually learned anything!? Basically looking at the delta isn't this just misleading? \nOk lets assume that on the last step you allow yourself to change eta to be non-zero. (I.e. we are just after the delta in theta, and what to show we can get the same delta in theta as EP which is how the proof is phrased). Then in that difference aren't you looking at s_{t+1} and s_t rather than s_{t+1} and s_0, which is what EP would do? In EP you have s^\\beta_* - s_*. This is not what you get if you don't update theta and apply C-EP? \n\nI think there might be something I'm missing about the mathematical argument here. \n\nAt a higher-level question, we talk about the transition function F as being a gradient vector field, i.e. there exist a phi such that F is d phi/d theta.  Why is this assumption biologically plausable ? Parametrizing gradient vector fields in general is far from trivial, and require very specific structure of the neural implementation of F to be true. Several works have looked at parametrizing gradient vector fields (https://arxiv.org/abs/1906.01563, https://arxiv.org/pdf/1608.05343.pdf) and the answer is that without parametrizing it by actually taking the gradient of a function there is not much of a choice. \nIncidentally, here we exploit that  F = sigma (Wx), with W symmetric. This is a paramtrization of a gradient vector field, i.e. of xU, where UU^T =W I think. But if you want to make F deep than it becomes non-trivial to restrict it to gradient vector field. Is the assumption that we never want to move away from vanilla RNNs? And W symmetric is also not biologically plausible. In C-EP you say is not needed to be symmetric, but that implicitly means there exist no phi and everything that follows breaks, no? \n\nI'm also confused by how one has access to d phi / ds and d phi / d theta. Again I feel like I'm missing information and the formalism is not introduced in a way that it is easy to parse. My understand is that you have an RNN that updates the state s. And the transfer function of this RNN is meant to be d phi / ds, which is trues if the recurrent weight is symmetric. Fine. But then why do we have access to d phi/ dtheta? Who is this function? Is the assumption that d s / dtheta is something we can compute in a biologically plausible way? Is this something that is obvious? \n \n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content: paper introduces a new variant of equilibrium propagation algorithm that continually updates the weights making it unnecessary to save steady states. T

Summary of discussion:
reviewer 1: likes the idea but points out many issues with the proofs. 
reviewer 2: he really likes the novelty of paper, but review is not detailed, particularly discussing pros/cons. 
reviewer 3: likes the ideas but has questions on proofs, and also questions why MNIST is used as the evaluation tasks.
Recommendation: interesting idea but writing/proofs could be clarified better. Vote reject.


Decision: Reject