user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Equilibrium Propagation with Continual Weight Updates

Paper content:
Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching those of Backpropagation Through Time (BPTT), but with a learning rule local in space.
Given an input x and associated target y, EP proceeds in two phases: in the first phase neurons evolve freely towards a first steady state; in the second phase output neurons are nudged towards y until they reach a second steady state.
However, in existing implementations of EP, the learning rule is not local in time:
the weight update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically.
This is a major impediment to the biological plausibility of EP and its efficient hardware implementation.
In this work, we propose a version of EP named Continual Equilibrium Propagation (C-EP) where neuron and synapse dynamics occur simultaneously throughout the second phase, so that the weight update becomes local in time.
We prove theoretically that, provided the learning rates are sufficiently small, at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss given by BPTT (Theorem 1).
We demonstrate training with C-EP on MNIST and generalize C-EP to neural networks where neurons are connected by asymmetric connections.
We show through experiments that the more the network updates follows the gradients of BPTT, the best it performs in terms of training.
These results bring EP a step closer to biology while maintaining its intimate link with backpropagation.
A motivation for deep learning is that a few simple principles may explain animal intelligence and allow us to build intelligent machines, and learning paradigms must be at the heart of such principles, creating a synergy between neuroscience and Artificial Intelligence (AI) research.
In the deep learning approach to AI (LeCun et al., 2015) , backpropagation thrives as the most powerful algorithm for training artificial neural networks.
Unfortunately, its implementation on conventional computer or dedicated hardware consumes more energy than the brain by several orders of magnitude (Strubell et al., 2019) .
One path towards reducing the gap between brains and machines in terms of power consumption is by investigating alternative learning paradigms relying on locally available information, which would allow radically different hardware implementations: such local learning rules could be used for the development of extremely energy efficient learning-capable hardware.
Investigating such bioplausible learning schemes with real-world applicability is therefore of interest not only for neuroscience, but also for developing neuromorphic computing hardware that takes inspiration from information-encoding, dynamics and topology of the brain to reach fast and energy efficient AI (Ambrogio et al., 2018; Romera et al., 2018) .
In these regards, Equilibrium Propagation (EP) is an alternative style of computation for estimating error gradients that presents significant advantages (Scellier and Bengio, 2017) .
EP belongs to the family of contrastive Hebbian learning (CHL) algorithms (Ackley et al., 1985; Movellan, 1991; Hinton, 2002) and therefore benefits from an important feature of these algorithms: neural dynamics and synaptic updates depend solely on information that is locally available.
As a CHL algorithm, EP applies to convergent RNNs, i.e. RNNs that are fed by a static input and converge to a steady state.
Training such a convergent RNN consists in adjusting the weights so that the steady state corresponding to an input x produces output values close to associated targets y.
CHL algorithms proceed in two phases: in the first phase, neurons evolve freely without external influence and settle to a (first) steady state; in the second phase, the values of output neurons are influenced by the target y and the neurons settle to a second steady state.
CHL weight updates consist in a Hebbian rule strengthening the connections between co-activated neurons at the first steady state, and an anti-Hebbian rule with opposite effect at the second steady state.
A difference between Equilibrium Propagation and standard CHL algorithms is that output neurons are not clamped in the second phase but elastically pulled towards the target y.
A second key property of EP is that, unlike CHL and other related algorithms, it is intimately linked to backpropagation.
It has been shown that synaptic updates in EP follow gradients of recurrent backpropagation (RBP) and backpropagation through time (BPTT) (Ernoult et al., 2019) .
This makes it especially attractive to bridge the gap between neural networks developed by neuroscientists, neuromorphic researchers and deep learning researchers.
Nevertheless, the bioplausibility of EP still undergoes two major limitations.
First, although EP is local in space, it is non-local in time.
In all existing implementations of EP the weight update is performed after the dynamics of the second phase have converged, when the first steady state is no longer physically available.
Thus the first steady state has to be artificially stored.
Second, the network dynamics have to derive from a primitive function, which is equivalent to the requirement of symmetric weights in the Hopfield model.
These two requirements are biologically unrealistic and also hinder the development of efficient EP computing hardware.
In this work, we propose an alternative implementation of EP (called C-EP) which features temporal locality, by enabling synaptic dynamics to occur throughout the second phase, simultaneously with neural dynamics.
We then address the second issue by adapting C-EP to systems having asymmetric synaptic connections, taking inspiration from Scellier et al. (2018) ; we call this modified version C-VF.
More specifically, the contributions of the current paper are the following:
• We introduce Continual Equilibrium Propagation (C-EP, Section 3.1-3.2), a new version of EP with continual weight updates: the weights of the network are adjusted continually in the second phase of training using local information in space and time.
Neuron steady states do not need to be stored after the first phase, in contrast with standard EP where a global weight update is performed at the end of the second phase.
Like standard EP, the C-EP algorithm applies to networks whose synaptic connections between neurons are assumed to be symmetric and tied.
• We show mathematically that, provided that the changes in synaptic strengths are sufficiently slow (i.e. the learning rates are sufficiently small), at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss obtained with BPTT (Theorem 1 and Fig. 2 , Section 3.3).
We call this property the Gradient Descending Dynamics (GDD) property, for consistency with the terminology used in Ernoult et al. (2019) .
• We demonstrate training with C-EP on MNIST, with accuracy approaching the one obtained with standard EP (Section 4.2).
• Finally, we adapt our C-EP algorithm to the more bio-realistic situation of a neural network with asymmetric connections between neurons.
We call this modified version C-VF as it is inspired by the Vector Field method proposed in Scellier et al. (2018) .
We demonstrate this approach on MNIST, and show numerically that the training performance is correlated with the satisfaction of Gradient Descending Dynamics (Section 4.3).
For completeness, we also show how the Recurrent Backpropagation (RBP) algorithm of Almeida (1987) ; Pineda (1987) relates to C-EP, EP and BPTT.
We illustrate the equivalence of these four algorithms on a simple analytical model ( Fig. 3 ) and we develop their relationship in Appendix A.
Equilibrium Propagation is an algorithm that leverages the dynamical nature of neurons to compute weight gradients through the physics of the neural network.
C-EP embraces simultaneous synapse and neuron dynamics, resolving the initial need of artificial memory units for storing the neuron values between different phases.
The C-EP framework preserves the equivalence with Backpropagation Through Time: in the limit of sufficiently slow synaptic dynamics (i.e. small learning rates), the system satisfies Gradient Descending Dynamics (Theorem 1).
Our experimental results confirm this theorem.
When training our vanilla RNN with symmetric weights with C-EP while ensuring convergence in 100 epochs, a modest reduction in MNIST accuracy is seen with regards to standard EP.
This accuracy reduction can be eliminated by using smaller learning rates and rescaling up the total weight update at the end of the second phase (Appendix F.2).
On top of extending the theory of Ernoult et al. (2019) , Theorem 1 also appears to provide a statistically robust tool for C-EP based learning.
Our experimental results show as in Ernoult et al. (2019) that, for a given network with specified neuron and synapse dynamics, the more the updates of Equilibrium Propagation follow the gradients provided by Backpropagation Through Time before training (in terms of angle in this work), the better this network can learn.
Our C-EP and C-VF algorithms exhibit features reminiscent of biology.
C-VF extends C-EP training to RNNs with asymmetric weights between neurons, as is the case in biology.
Its learning rule, local in space and time, is furthermore closely acquainted to Spike Timing Dependent Plasticity (STDP), a learning rule widely studied in neuroscience, inferred in vitro and in vivo from neural recordings in the hippocampus (Dan and Poo, 2004) .
In STDP, the synaptic strength is modulated by the relative timings of pre and post synaptic spikes within a precise time window (Bi and Poo, 1998; 2001) .
Each randomly selected synapse corresponds to one color.
While dashed and continuous lines coincide for standard EP, they split apart upon untying the weights and using continual updates.
Strikingly, the same rule that we use for C-VF learning can approximate STDP correlations in a rate-based formulation, as shown through numerical experiments by .
From this viewpoint our work brings EP a step closer to biology.
However, C-EP and C-VF do not aim at being models of biological learning per se, in that it would account for how the brain works or how animals learn, for which Reinforcement Learning might be a more suited learning paradigm.
The core motivation of this work is to propose a fully local implementation of EP, in particular to foster its hardware implementation.
When computed on a standard computer, due to the use of small learning rates to mimic analog dynamics within a finite number of epochs, training our models with C-EP and C-VF entail long simulation times.
With a Titan RTX GPU, training a fully connected architecture on MNIST takes 2 hours 39 mins with 1 hidden layer and 10 hours 49 mins with 2 hidden layers.
On the other hand, C-EP and C-VF might be particularly efficient in terms of speed and energy consumption when operated on neuromorphic hardware that employs analog device physics (Ambrogio et al., 2018; Romera et al., 2018) .
To this purpose, our work can provide an engineering guidance to map our algorithm onto a neuromorphic system.
Fig.
5 (a) shows that hyperparameters should be tuned so that before training, C-EP updates stay within 90
• of the gradients provided by BPTT.
More concretely in practice, it amounts to tune the degree of symmetry of the dynamics, for instance the angle between forward and backward weights -see Fig. 4 .1.
Our work is one step towards bridging Equilibrium Propagation with neuromorphic computing and thereby energy efficient implementations of gradient-based learning algorithms.
A PROOF OF THEOREM 1
In this appendix, we prove Theorem 1, which we recall here.
Theorem 1 (GDD Property).
Let s 0 , s 1 , . . . , s T be the convergent sequence of states and denote s * = s T the steady state.
Further assume that there exists some step K where 0 < K ≤ T such that s * = s T = s T −1 = . . . s T −K .
Then, in the limit η → 0 and β → 0, the first K normalized updates in the second phase of C-EP are equal to the negatives of the first K gradients of BPTT, i.e.
A.1
A SPECTRUM OF FOUR COMPUTATIONALLY EQUIVALENT LEARNING ALGORITHMS Proving Theorem 1 amounts to prove the equivalence of C-EP and BPTT.
In fact we can prove the equivalence of four algorithms, which all compute the gradient of the loss:
1. Backpropagation Through Time (BPTT), presented in Section B.2,
2. Recurrent Backpropagation (RBP), presented in Section B.3,
3. Equilibrium Propagation (EP), presented in Section 2,
4. Equilibrium Propagation with Continual Weight Updates (C-EP), introduced in Section 3.
In this spectrum of algorithms, BPTT is the most practical algorithm to date from the point of view of machine learning, but also the less biologically realistic.
In contrast, C-EP is the most realistic in terms of implementation in biological systems, while it is to date the least practical and least efficient for conventional machine learning (computations on standard Von-Neumann hardware are considerably slower due to repeated parameter updates, requiring memory access at each time-step of the second phase).

assistant: text: We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.