The least-control principle for local learning at equilibrium
Authors: Alexander Meulemans
Nicolas Zucchet
Seijin Kobayashi
Johannes von Oswald
Joao Sacramento
Venue: NeurIPS 2022 Accept (2022)
Keywords: Biologically-plausible learning
local learning rules
predictive coding
neuroscience
equilibrium recurrent neural networks
meta learning
deep equilibrium models
implicit models
optimal control

Abstract: Equilibrium systems are a powerful way to express neural computations. As special cases, they include models of great current interest in both neuroscience and machine learning, such as deep neural networks, equilibrium recurrent neural networks, deep equilibrium models, or meta-learning. Here, we present a new principle for learning such systems with a temporally- and spatially-local rule. Our principle casts learning as a \emph{least-control} problem, where we first introduce an optimal controller to lead the system towards a solution state, and then define learning as reducing the amount of control needed to reach such a state. We show that incorporating learning signals within a dynamics as an optimal control enables transmitting activity-dependent credit assignment information, avoids storing intermediate states in memory, and does not rely on infinitesimal learning signals. In practice, our principle leads to strong performance matching that of leading gradient-based learning methods when applied to an array of problems involving recurrent neural networks and meta-learning. Our results shed light on how the brain might learn and offer new ways of approaching a broad class of machine learning problems.

The least-control principle for local learning at equilibrium
Alexander Meulemans" 11, Nicolas Zucchet' Seijin Kobayashi Johannes von Oswald' João Sacramento
Department of Computer Science, ETH Zürich Institute of Neuroinformaties, University of Zürich and ETH Zürich (ameulema, nzucchet, seijink, voswaldj, rjoao] Gethz. ch
Abstract
Equilibrium systems are a powerful way to express neural computations. As special cases, they include models of great current interest in both neuroscience and machine learning, such as deep neural networks, equilibrium recurrent neural networks, deep equilibrium models, or meta-learming. Here, we present new principle for learning such systems with a temporally- and spatially-local rule. Our principle casts learning as least-control problem, where we first introduce optimal controller to lead the system towards solution state, and then define learning as reducing the amount of control needed to reach such state. We show that incorporating learning signals within dynamics as optimal control enables transmitting acrivity-dependen credit assignment information, avoids storing intermediate states in memory, and does not rely on infinitesimal learning signals. practice, our principle leads to strong performance matching that leading gradient-bas based learning methods when applied to array of problems involving recurrent neural networks and meta-learning. Our results shed light on how the brain might learn and offer new ways of approaching broad class of machine learning problems.
Introduction
The neural networks of the cortex are both layered and highly recurrent [1, 2]. Their high degree of recurrence and relatively low depth stands in contrast to the prevailing design of artificial neural networks, which have high depth and little to no recurrence. This discrepancy has triggered a recent of research into recurrent networks that more brain-like and which achieve high performance in perceptual tasks [3-7]. Concurrently, another line of recent work has shown that repeating short sequence of neural computations until convergence can lead to large gains in efficiency, reaching the state-of-the-att in various machine learning problems while reducing model size [8-10].
As develop models that come closer cortical networks by way of their recurrence, the precise mechanisms supporting learning in the brain remain largely unknown. While gradient-basee methods currently dominate machine learning, standard methods for efficient gradient computation result in non-local rules that are hard to interpret in biological terms [11, 12). This issue is particularly aggravated when applying these methods complex systems involving recurrence [13]. Indeed, while multiple interesting proposals [14-23] have emerged for how to efficiently compute loss function gradients for feedforward networks in piologically-plasse
ways, apart from a few notable exceptions [24, 25] much less progress has been made for recurrent networks. Furthermore, the majority of these methods requires that error feedback does influence network activity, property
"Equal contribution
Work partly done the Institute of Neuroinformaties, University of Zürich ETH Zürich
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

at odds with many experimental findings on on the problem of learning such recurrent systems using biologically-plassbbee activity-dependent local rules. To make progress in this longstanding question we make the assumption that our system at equilibrium, and formalize learning as the following optimization problem:
plasticity [26]. In this paper, we focus
(1) where I are the parameters we wish to learn, L is loss function which measures performance, and ((, 0) is the system dynamics which is an equilibrium point Φ. Our model is thus implicilly defined through a dynamics that is at equilibrium. Expressing our problem in this general form allows us model very broad class of learning systems, without being restricted neural network [27-29]. In particular, we cover both feedforward and recurrent neural architectures. More generally. f is not even restricted being a neural dynamics. Consider the case where f defines a learning algorithm governing the dynamics of the weights of a network; in this case (1) becomes a meta-learning problem, where the goal is to tune the pareta-parameterss of learning algorithm such that its performance improves.
ming L(Ø") s.t. f()",0)= = O,
partieular type of
Inspired by recent control- -bsed learning method for feedforward neural networks [30], we present a new principle for solving problems of the form (1) which yields learning rules that are (i) gradient- following, (ii) local in space, (iii) local in time, and in particular do not require storing intermediate states, (iv) deiiitt-deeeedeett by embedding error information into the network activity, and (v) not reliant infinitesimally small learning signals. To meet all five criteria at once, we depart from direct gradient-bas based optimization of the loss, and we reformulate gradient-based learning within the framework of optimal control as problem of least-control.
Such problems can be approached in two steps. First, an optimal controller provides additional feedback input leading the dynamical system to least-control state: equilibrium point in which the loss L is minimized with the least amount of control. Subsequently, the parameters are changed to further reduce the amount of control at the resulting equilibria. Critically, this minimization can be achieved using a local gradient-base based rule for which all the required information is available at the controlled equilibria. This should be contrasted to the ubiquitous backpropagation of error algorithm [31, 32] (and its variant for equilibrium systems [33, 34) which yields non-local parameter updates based on the outcome of two separate phases. Importantly, our least-control problem is intimately related to the original learning problem; we make the connection between the two precise by identifying mathematical conditions under which least-control solutions correspond to minima of the original objective L(Ø"), or bound of it.
We apply our principle learn equilibrium neural networks featuring variety of architectures, including both connecco and convolutional feedfoward networks, -oteeeely-oonnected recurrent networks of leaky integrator neurons, as well as deep equilibrium models [8-10], recent family of high-performance models which repeat until convergence a sequence of complex computations. To further demonstrate the generality of our principle, learning problem where the goal is to change the internal state of complex synapse such that future learning performance is improved [35]. We find that our single-phase local learning rules yield highly competitive performance in both application domains. Our results extend the limits of what can be achieved with local learning rules, opening novel avenues to an old problem.
then consider recently studied meta-
2
The least-control principle
2.1
The principle
We introduce the least-contro principle for learning dynamical systems that evolve in time according to é(1) = ((((), 0) until an equilibrium o² is reached. Instead of directly minimizing a loss evaluated at equilibrium L(ø") as in (1), we augment the dynamics dtt) = ((((), 0) + +()) with control signal (t) that drives the system towards a new controlled equilibrium where the loss is
minimized (c.f. Figure 1). Our principle recasts learning as minimizing the amount of control needed
to reach that state:
min 0.0.0
| 2
s.t. f( + w= = O, VoL(O) = 0.
(2)
The two constraints in the equation above ensure that we reach steady state of the controlled dynamies, and that the loss I. is minimized at this state. Without loss of generality, we consider a single data point above (c.f. Section S5 in the supplementary materials).

---

A
I(6)
B
I(Ø)
C
y
controller
[(. 0)
ò 1(0,0")
y target
à f(Ø,0) 0) +
X = -(0,00) (00+0++0 +
1(0..0) 0)
11
91
Figure I: (A, B) Intuition behind the least-control principle. (A) During learning, the free dynamics (dark blue) is augmented with a controller (light blue) that drives the system towards an equilibrium that minimizes the loss function (grey, the darker the smaller the loss is). Our principle prescribes minimizing the amount of optimal control sh. (light green) needed at an equilibrium that minimizes L. (B) After learning, the free equilibrium coincides with the controlled equilibrium and hence minimizes the loss L. (C) Example of an instantiation of the principle for the supervised learning of a feedforward neural network. The controller both drives the output units y of the network towards target value yfarget and influences the rest of the network. The control signal 3 can be computed directly from the output error or indirectly by leveraging internal feedback within the network.
To solve (2), we first fix the parameters 0. We then find an optimal control v/. and controlled equilibrium oh that minimize the following least-control objective, for the current setting of
1 1l 2
min 0.0
s.L. f(b,0) 0, V V = 0.
We call o, the least-control state and th. the least control. Here, vi represents an optimal control with terminal cost 1| and terminal constraint enforcing that the optimally controlled equilibrium 6, minimizes the loss L. Critically, both credit assignment and system dynamics are now combined into single phase through the optimal control v/., We then take a gradient-following update in @ on H(0) := 2 1l, the lcast-control objective of Eq. 3, which implicitly depends on through vl..
Theorem 1 shows that the optimal control tb. contains enough information about the learning objective so that the gradient easy to compute. We prove Theorem along with all the theoretical results from Section 2, in Section S2. We use 0, to denote partial derivatives w.r.t. the variable x.
Theorem (Informal). Under some mild regularity conditions, the least-control principle yields the following gradient for 0:
VoH(0) = -d. v. = ()) f) (((
(4)
To illustrate what our principle achieves, let us briefly consider the setting where f(Ø, 0) implements a feedforward neural network, typically learned by backpropagation of a supervised error signal. In this case, the variable & corresponds to the state of the neurons (e-g. their firing rate or postsynaptic voltage) and 0 to the synaptic connection weights. The least control th, then drives the network to an equilibrium that minimizes the output loss, while being of minimum norm. The resulting weight update (c.f. Section S6.2) now represents a local Hebbian rule multiplying presynaptic input with postsynaptic control signal: the neural activity implicitly encodes credit assignment information, in such way that local learning becomes possible. By contrast, in the backpropagation algorithm, errors do not directly influence neural activity and are computed in a separate phase.
The simplicity of our parameter update becomes apparent when contrasting it with the gradient associated with the original objective (1), as the implicit function thcorem (c.f. Section S3) gives
VoL()") = f (6*,,) B)T ;)T
(0+f("",0) - [('(
(5)
Calculating this gradient directly requires inverting the Jacobian 0gf(ó", 0). which intractable for large-scale systems. The standard way of dealing with this issue is to resort to the two- wwo-phase recurrent backpropagation algorithm, which estimates VoL(")) by running second linear dynamics until equilibrium, while holding the first equilibrium state o² in memory. For acyclic computation graphs, of which feedforward neural networks are a prime example, simplifies into the error backpropagation algorithm. Those procedures are still more complicated than ours: they require implementing a specialized auxiliary dynamical system, holding intermediate states memory, and alternating between inference and gradient computation phases.
3

---

2.2 A general class of dynamics leads to the same optimal control and parameter update
Our principle is agnostic to the dynamics (t) of the controller; the only assumption made is that its value ei, at the steady state optimal, i.. the = 4... We use the subscripts denote equilibrium states of the controlled dynamics and to indicate optimality on the least-control objective of Eq. 3, and the superseript for a free (not controlled) equilibrium. In particular, the parameter update only depends on information available at the controlled equilibrium state (ø., v..). Here, we capitalize on this important flexibility by designing different simple feedback controller dynamics that lead the same optimal steady state. We consider feedback controllers which take the following general form:
à f 0) + Q(0, 0)u, and ù = - VVII(0) ax
(6)
where u controller defined on the output units and Q( 8) determines the influence of the controller on the entire system. We note that here and throughout our paper we omit the dependence on time (above, of (() and u(t) from our equations to unclutter the presentation.
(6), the mapping Q(ú 0) can either be direct mapping from the output controller u to the entire system, or indirect one leveraging internal feedback mechanisms in the system (c.f. Figure IC). The controller u may be seen leaky integral controller with leak rate a. that uses the output error V to drive the network towards a minimal-loss state (c.f. Section S2), Returning to our example of a neural network, leads the network into state configuration where the output neurons y at the target value y targget when a = 0.
We use leaky integral control in (6) for explanatory purposes and as we use it in our experiments. We remark however that our theory goes beyond this form of output control; in particular, generalizes to any controller satisfying aur + VoL(o.) = 0 at equilibrium. For example, the proportional control - V & (o) satisfies this condition for a = 3-1
Theorem 2 provides general conditions for the dynamics (6) to converge an optimal control signal needed for obtaining first-order parameter updates in Theorem 1.
Theorem 2. Let (Φ,,u,, be steady state of the generalized dynamics (6), Assume that 0zf(e..0)
is invertible and that the following column space condition is satisfied at equilibrium:
col (ol 0)] = row [[0, Id] f 6)=
where the matrix [0, Id] has (δ) columns and as many rows as the system has output units. Then, 4/. = =(Ø,, is an optimal control for the least-control objective (3), in the limit a + 0.
Interestingly, Theorem 2 shows that many different feedback mappings from the output controller towards the system state & are possible, as long as the mapping (é,,,) 0) satisfies the column space condition (7) at equilibrium, in the limit a + O. Furthermore, the control & need not be instantaneous and can have its own temporal dynamics alongside the output controller dynamics ú, as the column space condition (7) is only defined at equilibrium. We exploit this property and construct a general-purpos algorithm that computes an optimal control vo.:
f(6 8) + 16, and sb = ()
with the leaky integral controller defined in (6). This inversion dynamics has two special properties. First, satisfies the conditions of Theorem 2 by construction (c.f. Section S2.3). Second, as defined above, e resembles the second-phase dynamics of the (recurrent) backpropagation algorithm (c.f. Section S3), or in case of feedforward networks, the layerwise backpropagation of error signals in the error backpropagation algorithm. Intuitively, in this case we find an optimally-controlles equilibrium by simultaneously (in a single phase) running the first- and second-phase dynamics of (recurrent) backpropagation, with the critical difference that here the dynamical equations of both phases interact through the relation = f(Ø, 0) + +
2.3
The least-control principle solves the original learning problem
Iterating our parameter update minimizes the least-control objective H(() and not directly the original objective of learning (1). The latter is however the objective of ultimate interest, used to measure the performance of the system after learning, when it is no longer under the influence of a controller. We now show that there a close link between the two objectives and identify a wide range of conditions under which least-control solutions also solve the original learning problem (1).

---

Proposition 3 shows the intuitive result that if the optimal control is minimized to zero, the equilibria of the controlled and free dynamics coincide and hence the loss L is minimized at the free equilibrium as well (c.f. Figure 1.B).
Proposition 3. Assuming L is convex on the system output, we have that the optimal control tb, is equal to 0 isf. the free equilibrium o² minimizes L.
As v.. - = f), reaching v. = 0 can only be done if the model is powerful enough to perfectly fit the controlled equilibria for all data points. We show in Proposition 4, that overparameterization i.e, having more parameters 0 than system states o, indeed helps solve the original learning problem (I). The condition 0g ( 0) being of full row rank, that is needed this proposition, can only be satisfied when the dimensions satisfy [0) > (é), with & the concatenated states of all data points, hence providing another perspective on why overparameterization helps.
Proposition 4. Assuming L is convex on the system output, a local minimizer O of the least control objective H. is a global minimizer of the original learning problem (1), under the sufficient condition that f 8) has full row rank.
Fully minimizing the amount of control to zero is not always possible, which highlights the need to understand the relation between the least-control objective H()) = 1 v/ 1 and the loss ( from the original learning problem (1). Proposition 5 shows that we can do so, as minimizing ||w.11?
indirectly minimizes the loss L(d*), under some regularity and strong convexity assumptions.
Proposition S. |[ then
is p-strongly convex, L is M -Liischiitz continuous and the minimum of L is 0.
V/ V2M
V/ V2M (I O)I.
L(Ø)) S
Approximate equilibria. Our least-control theory requires the system to be at equilibrium, and that the loss minimized at the controlled equilibrium. This almost never the case in practice, for instance because the dynamics are run for finite amount of time, the column space condition for Q not perfectly satisfied, or because take therefore does not strictly follow the gradient of the least-control objective H(O). We formally show in Section S2.6 that our update resulting from an approximate optimally-controllen equilibrium is close to the gradient V. More precisely, under some regularity conditions on f. the error in the gradient estimation is upper bounded by the distance between d. and its estimate. Additionally, if we reach equilibrium but is non-zero the update ( 0) T v.. follows the gradient of an objective very similar to the least-control one H. under strict conditions on the feedback mapping Q.
not zero the output controller u dynamics. The update we
2.4 The least-control principle as constrained energy minimization
We now provide a dual view of our least-control principle from the perspective of energy minimization. This will allow designing a second general-purpose dynamics for computing the least control v/., and establishing a link to another principle for learning known as the free-energy principle. To arrive at an energy function, we rewrite the least-control objective (3) by substituting the constraint into this objective, leading to the following constrained optimization problem:
3. arg min& 1 (o, (1)0))2 s.t. VoI(o)= 0
The role of optimal control can therefore be reinterpreted as finding the state that is the closest to equilibrium under the free dynamics, among the states that minimize the loss function.
Next, we introduce the augmented energy F(ó, 0, 3) = (0, potential to the equilibrium energy () 0)1? and take the perfect control limit, B - co, where this potential dominates the energy. As we show in Section S2.5, minimizing the augmented energy with respect to the state variable @ leads to fulfilling our objective (9). Interestingly, when 3 - oo and f( 0) implements a feedforward neural network, we recover the free-energy function which governs the predictive coding model of Whittington and Bogacz (36), which was obtained from an entirely different route based on variational expectation maximization for a probabilistic latent variable model [37] (c.f. Section S4).
which adds nudging
Energy-based dynamics as optimal control. We leverage this connection design second ceneral-purposs dynamics that computes an optimal control, by using the gradient flow on F:
à -0gf(0 8)T f( 0) LQ)
(10)
5

---

Here, the control A is implicitly contained in the dynamics of Φ. Part of the dynamics, (()) plays the role of an mnfiitely--strogg proportional controller on the system output; another part is in charge of optimally sending the teaching signal from the output back to the rest of the system. We refer to Section S2.5 for more details.
Two opposing limits: perfect control vs. weak nudging, The update 0p.f f that we obtain in Theorem 1 has been extensively studied in the weak nudging limit 3 - [36, 38, 39], which sits at the opposite end of the 3-spectrum of the perfect control limit studied here. A seminal result shows that as 3 + the gradient (o"( of the original objective function is recovered [24, 40]. However, this update is known to be sensitive to noise as the value of f can be very small at the weakly- equilibrium [35]. Our least-control principle works at the opposite perfect control end of the spectrum (5 + oc), and it is therefore more resistant to noise. One of our main contributions to show that using the same update, but instead evaluated at an optimally- optimally-controlled state e.. still performs principled gradient-based optimization, however now on the least-control objective HH()).
3 Applications of the least-control principle
The least-control principle applies to a general class of dynamical systems equilibrium. By contrast, the majority of previous work on local learning focused on designing circuits and rules tailored towards feedforward neural networks [14, 17, 23, 36, 41-45]. We now demonstrate the generality of our principle by applying it to two problems of interest in neuroscience and machine learning: deep and recurrent neural network learning, and meta-learning. 3 In both cases, principle leads to activity-depende local learning rules by leveraging Theorem 1, while making use of simple and flexible optimal controllers to feed back credit assignment information (c.f. Theorem 2). We test our learning and meta-learning systems on standard benchmarks and find that they perform competitively when compared to conventional gradient-based methods.
3.1 The least-control principle for feedforward and recurrent neural networks
As a first demonstration of learning according to our principle we consider feedforward neural networks with multiple layers of hidden neurons, the current preferred architecture for a large array of perceptual problems. Moreover, prior work on sealooccally-plassibte alternatives to backpropagation has mostly focused on feedforward networks, making them a natural first choice to study our principle.
Motivated by the massive recurrence of cortical networks, then turn to equilibrium recurrent neural networks as a second application of our principle, aiming to demonstrate its applicability beyond feedforward models. While there is debate over the precise functional roles of recurrent processing in the brain, there is strong experimental evidence that these connections are not limited to transmitting learning signals and play an active role in information processing [e.g.. 46-49]. From the viewpoint of machine learning, it has been shown that equilibrium RNNs are often more parameter-efficien than feedforward networks, while reaching state-of-the performance [3, 7, 9].
More concretely, we consider a generic neural network driven by a fixed input x assumed to converge to an equilibrium state, obeying the following free dynamics (e.g. Figure 1.C):
6 f 6) = - W Wo(Ø) + Ux
(11)
with ø the neural activitics, W the internal synaptic weight matrix, U the input weight matrix and the activation function. Note that this notation is general enough to encompass feedforward and recurrent network architectures; while freely chosen W and U yield a vanilla RNN, structuring W and U appropriately (with block-diagona structure) allows recovering standard feedforward fully-connected or convolutional deep neural networks. We point to Section S6 for full architectural details. The learning problem then consists of optimizing the network weights 0 {W, U} on a loss L(Ø*) at the equilibrium activity 6² This loss is only measured on the output neurons, a subset of ø.
We now add control signal v to the dynamics (11). Theorem 1 guarantees that if the controlled steady state dh. is optimal, the following updates minimize the least-control objective H()):
AW (o.) and AU = T.
(12)
To showcase the flexibility of our principle, we use Theorem 2 to design various simple feedback circuits that compute these control signals, as we detail next. We visualize those circuits in Figure $2
Code for all experiments is available at https: : /tthgtgttthhu....//. www.///eeinoooo//eeooooooooo gebbayashi/eeastcooe

---

3.1.1 A simple controller with direct linear feedback
To instantiate the least-control principle in its simplest form, we project the output controller u onto the hidden neurons & with direct linear feedback weights Q, which a generalization of a recent feedforward network learning method [30] to equilibrium neural networks. Broadcasting output errors directly [50] may be seen as one of the simplest possible ways avoiding the weight transport problem [11, 12, 16]. This results in the following dynamics:
- + W (Ø) + Ux + Qu, and ú V (b) auu
(13)
with u simple leaky integral output controller (c.f. Section 2.2), and V v I. the output error.
error.
Theorem guarantees that Qu, is an optimal control in the limit of a + 0, if the feedback weights for all data samples.
Q satisfy the column space condition col[Q] = row [[0, [dd - (oo()))) For a linear feedback mapping, this condition cannot be exactly satisfied for multiple samples as the row space is data-dependent, whereas Q is not. Still, we learn the feedback weights Q to approximately satisfy this column space condition, by using a local Hebbian rule that operates simultaneously the learning of the other weights, inspired by recent work on feedforward networks [30, 51] (details in Section S6). The resulting update provably finds feedback weights Q that satisfy the column space condition for one sample, and we empirically verify that the control signal it gives in the multiple samples regime is close to optimal (c.f. Section S6).
Despite the simplicity of this linear feedback controller, show that the training procedure described above is still powerful enough to learn both a -vo-hideen-layer (each of 256 neurons) fully-connected feedforward network and an equilibrium RNN with fully-connected recurrent layer of 256 neurons on the MNIST digit classification task [52] (c.f. Table 1, LCP-LF). Notably, it performs almost as well as backpropagation (BP) and recurrent backpropagation (RBP), the current methods of choice for equilibrium RNN training. We use fixed point iterations to find the steady states Φ and u, in Eq. (13) for computational efficiency (c.f. Section S6 for more simulation details). We also observe that the empirical performance improves by changing the parameter updates to AW = a' (dt))(u (Q.. and AU = d' 'Qu,x² I when the column space condition is not perfectly satisfied, corroborating the findings of Meulemans et al. [30, 51] for feedforward neural networks.
Table 1: MNIST test set classification accuracy (%) for feedforward network (2x256 neurons) and an RNN (256 neurons) trained by resp. backpropaga- tion (BP) and recurrent backpropagation (RBP), and by the least-control principle (LCP) with linear feed- back (LF), energy-based dynamics (EBD), dynamic inversion (DI) and with learned feedback weights (DI KP). Mean std computed over 3 seeds.
Method
FF
RNN
LCP-LF
97.73 10.07
97.70
LCP-DI
98.11 =0.07
97.58++0.12
LCP- DI (KP)
98. 00000
97.75
LCP-EBD
98.00=0.03
97.60
BP/RBP
98.29-0.11
97.87#0.19
3.1.2 Learning with general-purpose optimal control dynamics
In the previous section we designed and tested a simple error broadcast circuit and showed that we can learn it such that the conditions of Theorem 2 are approximately met. This approximate feedback is likely insufficient for more complex tasks which require hamnsssin depth or strongly- dynamics [53]. This leads us to consider two feedback circuits with more detailed architectures, which have the capacity to steer the network to an (exact) optimally-controlle equilibrium.
The first circuit is based on our general-purpose dynamic inversion (8). Applied to RNNs it yields:
ò = - + Wo( + Ux + tb and ف = - + S + u.
(14)
The second circuit exploits the dual energy minimization view of our principle presented in Section 2.4. Taking the energy- -erregdinenniingggg dynamics á = =-a ,, 3) of Eq. 10, we obtain
e - + o' (b)) u, and + - - Wa(
(15)
where we use the output controller u of Eq. 6 as generalization of the proportional control -BV, L to make the two circuits more directly comparable. Above, we introduce decoupled feedback weights S to avoid sharing weights between & and th and we learn S to align with W by using the Kolen- Pollack local learning rule [42, 54], which simply transposes the weight update AW (12) and adds weight decay to both update rules: AS o(o)) v - -7S.
7

---

Both circuits implement optimal control through neural dynamics, avoiding explicit matrix inverses and phases. In biological terms, they lead different interpretations and implementation possibilities, highlighting the flexibility of our learning principle. The dynamic inversion circuit (14) fits naturally with dendritic error implementations, where v can be linked to feedback dendritic compartments whose signals steer plasticity [14, 17, 20, 23, 30, 41, 51, 55J. In particular, by leveraging burst- multiplexing circuits [14, 23] or interneuron microcircuits [17], all information for the dynamics and weight updates can be made locally available (cf. Section S6). On the other hand, the energy- minimizing dynamics (15) naturally leads to an implementation based on prediction and error neuron populations, characteristic of predictive coding circuits [36, 56 58). We provide a high-level discussion of the two circuit alternatives (14, 15) in Section S6.
First, we test our optimally controllee neural networks by training a fully-connected equilibrium RNN on MNIST with circuit (14), while varying the recurrent layer width, cf. Figure 2. This analysis shows that our theoretical results of Section 2.3 translate to practice, as for strongly overparameterized networks, both H and L are mini- mized to zero, whereas for underparameterrized networks, minimizing the least-control objective (())) leads to a sig- nificant decrease in L. Moreover, our feedforward (2x256 hidden neurons) and recurrent (256 hidden neurons) net- works achieve competitive test performance compared to backpropagation (recurrent backpropagation, for RNNs), both when the feedback weights are trained (LCP-DI KP) and when they are tied (S = : LCP-DI and LCP-EBD), irrespective of whether we use the energy- based (15) or dynamic inversion (14) optimal control dynamics, cf. Table 1.
LCP amount control
1.5
Amount of control (x10P)
Train (x=10)
LCP train loss
1.0
REP train loss
1.0
0.5
10" Recurrent ley wisth
Figure 2: Training loss L and the amount
of control |1 ll = V2H after training un- till convergence on MNIST with various recurrent layer widths.
Next, we test whether the circuits and rules obtained through our principle scale to more complex architectures and tasks and move to the CIFAR- 10 image classification benchmark [60]. We study two distinct models: a standard feedforward convolu- tional neural network, and deep equilib- rium model (DEQ) consisting of a recur- rent block of convolutions [9]. The latter can be interpreted as an efficient infinite- depth residual neural network with weight- sharing [61, 62]. Full architectural details are provided in Sections S6.4.3 and S6.5.2: we note that to showcase our least-control principle as a scalable learning method in the DEQ experiments, we make a number of design choices that include using automatic differ- entiation to compute certain derivatives. For our feedforward network experiments we make no such concessions, and we further investigate weight-transsport-ffes variant using Kollen-Pollack learning. We show on Table 2 that the networks trained by our principle are able to match those trained with backpropagation. Interestingly, we observe that the DEQ models considered here are not overparameterized enough to not need control, but that this does not impair final predictive performance (c.f. Table S5).
Method
C10 (FF)
C10 (RNN)
INR
BP/RBP
77.58=0.14
80.14
25.47=4.16
LCP-DI
40.18
80.26
25.11
I.P-KP
77.16+0.10
Table 2: C10: CIFAR-10 test accuracy (%) of convolutional feedforward (FF) and equilibrium RNN (RNN), learned by dynamic inversion (DI; Kollen- Pollack learning without weight sharing: KP) and by standard/recurrent backpropagation (BP/RBP). INR: the peak signal-to-noise ratio (in dB) of an implicit neural representation learned on natural image dataset [59]. Mean + std computed over seeds.
Lastly, we turn to implicit neural representations [63], yet another task for which recurrent connections are known to be particularly beneficial [10]. The network there has to represent an image in its weights so that able to map pixel coordinates to their color values. Briefly, we once again find that our principle performs competitively to RBP. We refer full details in Section S6 for more details.
3.2 The least-control principle for meta-learning
The generality of the least-control principle enables considering learning paradigms other than supervised learning. Here, we show that it can be applied to meta-learning, a framework that
8

---

aims at improving learning algorithm by leveraging shared structure in encountered tasks [64]. This framework is gaining traction neuroscience, having been featured in recent work aiming at understanding which systems [65-67] and rules [35] could support meta-learning in the brain.
Meta-learning complex synapses. Following [35, 68), we model the learning algorithm for training a neural network with synaptic weights & on some task T as the minimization of a data-driven loss L Laain (o). regularized by a quadratic term Allø - wll2. This regularizer can be interpreted as a model of a complex synapse; similar models have been introduced in studies of continual learning and long- term memory [69-73]. In the context of meta- veta-learning. the fast weights & quickly learn how to solve a task and À determines how strongly Φ is pulled towards the slow weights w, which consolidate the knowledge that has been previously acquired. The capabilities of the learning algorithm are measured by evaluating - (d?), the performance of the learned neural network on data from task T that was held out during learning. Meta-learning finally corresponds to improving the learning performance on many tasks by adjusting the meta retaaparameters 0 = leading to the following problem:
X min E,, | [++)] s.t. x = arg min L Jearn (0) + 2 wll?.
(16)
0
Above, the parameters é no longer represent neural activities and @ synaptic weights as in the previous section; o and 0 w now designate synaptic weights and meta- perammarsmssers, respectively.
Meta-parameter updates. Using the first-order conditions -V -Vg ( \(Øō 0) O associated to oi being minimizer in the previous equation yields an optimization problem which the least-control principle can be applied.
Method
Test accuracy
FOMAML [74]
89,40=+0.50
T1-T2 [75]
89.72 10.43
Our principle prescribes running the controlled dynamics
Reptile [76]
89.43-10.14
LCP (ours)
91.00=00.24
3 = - V Φ I. learn
A(o w) + u,
(17) iMAML [68]
94,46+0.12
ú c eval ())
CML [35]
4..122
or any other process that finds the same fixed points (øI, w!). The theory we developed in Section 2.2 guarantees that v. is an optimal control as long as a - 0 and the Hessian of the evaluation loss Lo at the equilibrium oz is invertible. Hence, we do not need complex dynamics to compute the gradient, highlighting the flexibility and simplicity of learning according to the lcast-control principle. The resulting single-phase meta- meta-lcarning rule is local to every synapse (c.f. Section S7):
Table 3: Meta-leamingg neural net- works with complex synapses on Om- niglot 20-way 1-shot learning tasks. Mean + std test set classification ac- curacy (%) computed over 3 seeds.
Aw = Au
(18)
Experimental results. Our meta-learning algorithm stands out from competing methods as being both principled, first-order and single-phase. On the one hand, existing first-order and single-phase methods either approximate meta- centtaggadients (e.g.. FOMAML [74] and T1-T2 [75]) or rely on heuristics (e.g. Reptile [76]), and therefore do not provably minimize meta-objective function. On the other hand, full meta gradient descent methods require two phases to backpropagate through learning [74], invert an intractable Hessian [68, 77], learn slightly modified task [35], and often require second derivatives 168, 74, 77]. Here, we compare our rule to competing first-order methods (FOMAMI, T1-T2 and Reptile) and meta-gradient approximation algorithms that use the same synaptic model (implicit MAMI. and contrastive meta-learning: iMAMI. and CML. on the Omniglot few-shot visual classification benchmark [78]. Each Omniglot task consists of distinguishing 20 classes after seeing only 1 sample per class; we choose this 20-way 1-shot regime as the gap between first- and second- order methods is reputedly the largest compared to other variants of this benchmark. We find that our meta-learning rule significantly outperforms other single-phase first-order methods, while still lagging behind iMAML, second-order and two-phased method, and CML, a two-phased first-order method that can approximate the meta- mnea--grdientt arbitrarily well. These findings are particularly interesting since our synaptic model ( (16) in the underparameterized regime: there a separate set of weights Φ- per task 7 but only one set of consolidated weights w for the whole task distribution. The strong performance observed here is another demonstratation that least-control solutions can perform well even when overparameterization arguments (cf. Section 2.3) do not hold.
9

---

4 Discussion
Control theories of neural network learning. The relationship between control theory and neural network learning is an old and intricate one. is by now well known that backpropagation, the method of choice for computing gradients in deep learning, can be seen as the discrete-time counterpart of the adjoint method developed in control theory for continuous-time systems [79, 80). One of the early papers realizing this connection formulated neural network learning as an optimization problem under equilibrium constraints, as in the original problem statement (1), and rederived the recurrent backpropagation algorithm [81]. More recent work has explored using feedback neural control circuits which influence the neural dynamics of feedforward [30, 51] and recurrent [82-84] networks to deliver learning signals. Different from such approaches, our least-control principle is rooted in optimal control: instead of setting the control from ansatz, we derived it from first principles. This allowed us to arrive at a simple local rule for gradient- based learning, and to identify general sufficient conditions for the feedback control signals.
Predictive coding from the least-control perspective. We have shown that least-control problems correspond to free-energy minimization problems. This dual view of least-control problems allows connecting to influential theories of neural information processing based on predictive coding [19, 36, 37, 56, 58]. Our results elucidate predictive coding theories in at least three ways. First, Theorem I justifies the parameter update taken in supervised predictive coding networks [36] as gradient-based optimization of our least-control objective. This connection offers a novel view of the predictive coding dynamics as an optimal control algorithm in charge of controlling hidden neurons, while "clamping' output neurons to the respective targets can be seen as a form of output control. Previous arguments for gradient-base based learning were only known for the weak nudging limit (3 + 0) where learning signals are vanishingly small [24], which is problematic in noisy systems such the cortex [85]; our novel theoretical and experimental findings characterize the opposite perfect control (3 + oc) end of the spectrum. Second, Theorem broadens the class of neural network circuits which may be used to implement predictive coding, beyond the that have been proposed far [17, 36]. Finally, our experiments on recurrent neural networks using monneryy-miiiiziing dynamics to assign credit are to the best of our knowledge the first of the kind, and serve validate the effectiveness of predictive coding circuits beyond feedforward architectures.
microcireuits
Limitations. A limiting factor of our theory is that in its current form it is only applicable to equilibrium systems. While the study of neural dynamics at equilibrium is old endeavor in neuroscience and neural network theory [86, 87], it remains unclear how strong an assumption it is when modeling the networks of the brain. Extending our principle out-of-equilirrium dynamic is an exciting direction for future work. Another limitation concerning the breadth of problem (1) is that we did not allow the loss function L to directly depend on the parameters O. We discuss extensions to this more general case in Section S2.2, however not all loss functions L in this more general class satisfy the conditions needed obtain a local parameter update. Lastly, it should be noted that although our principle requires only one relaxation phase, its computational cost often exceeds the combined cost of performing the two phases of recurrent backpropagation (cf. Section S6.6).
Concluding remarks. We have presented a new theory that enables learning equilibrium systems with local gradient-based rules. Our learning rules are driven by changes in activity generated by an optimal control, in charge of feeding back credit assignment information into the system as evolves. Taken together, our results push the boundaries of what is possible with single-phase activity-dependent local learning rules, and suggest an alternative to direct gradient- based learning which yields high performance in practice.
Acknowledgments and Disclosure of Funding
This research was supported by an Ambizione grant (PZ00P3_1866) from the Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1) awarded to João Sacramento. Seijin Kobayashi was supported by the Swiss National Science Foundation (SNF) grant CRSII5_173721. Johannes von Oswald was funded by the Swiss Data Science Center (J.v... P18-03). We thank Angelika Steger, Benjamin Scellier, Walter Senn, Il Memming Park, Blake A. Richards, Rafal Bogaez, Yoshua Bengio, and members of the Senn, Bogacz and Richards labs for discussions.
10

---

References
[1] D. J. Felleman and D. C. van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1(1):1-47, 1991.
[2] Rodney J. Douglas and Kevan A.C. Martin. Neuronal circuits of the neocortex. Annual Review of Neuroscience, 27111:419-455) 2004.
[3] Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprini arXiv: 004.03640, 2016.
[4] Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L.. Yamins. Task-driven convolutional recurrent models of the visual system. Advances in Neural Information Processing Systems, 31, 2018.
[5] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. Yamins, and James J. DiCarlo. Brain-like object recognition with high-performing shallow recurrent ANNs. In Advances in Neural Information Processing Systems, 2019.
[6] Ruben S. van Bergen and Nikolaus Kriegeskorte. Going in circles is the way forward: the role of recurrence visual inference. Current Opinion Neurobiology, 15:176-993 2020.
[7] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models. In Advances in Neural Information Processing Systems, 2020.
[8] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 2019.
[9] Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 2020.
[10] Zhichun Huang, Shaojie Bai, and Zico Kolter. Implicit:: implicit layers for implicit represen- tations. In Advances in Neural Information Processing Systems, 2021.
[11] Stephen Grossberg, Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-63, 1987.
[12] Francis Crick. The recent excitement about neural networks. Nature, 337:129-1233 1989.
[13] Timothy P. Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology, 55:82 89, 2019.
[14] Konrad P Körding and Peter König. Supervised and unsupervised learning with two sites of synaptic integration. Journal of Computational Neuroscience, 11(3):207-155, 2001.
[15] Kong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target prop- agation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2015.
[16] Timothy P. Lillicrap. Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communi- cations, 711):13276, 2016.
[17] João Sacramento, Rui Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, 2018
[18] Pieter R. Roelfsema and Anthony Holtmaat, Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 1880 2018.
[19] James C. R. Whittington and Rafal Bogacz. Theories of error back pao-ppropagation in the brain. Trends in Cognitive Sciences, 23(3):235-250) 2019.

---

[20] Blake A. Richards and Timothy P. Lillicrap. Dendritic solutions to the credit assignment problem. Current Opinion Neurobiology, 54:28-33 36, 2019.
[21] Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepees, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lindsay, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zyybberberg Denis Therien, and Konrad Kording. A deep learning framework for neuroscience. Nature Neuroscience, 22(11): 1761-1770, 2019.
[22] Timothy P. Lillicrap. Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
[23] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature Neuroscience, 24(7):1010-1019) 2021.
[24] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, 2017.
[25] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. A solution the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11()):3625 2020.
[26] Stephen J. Martin, Paul D. Grimwood, and Richard G.M. Morris. Synaptic plasticity and memory: evaluation of the hypothesis. Annual Review of Neuroscience, 2311):649-711 2000.
[27] Brandon Amos. Differentiable optimization- modeling for machine learning. PhD Thesis, Carnegie Mellon University, 2019.
[28] Stephen Gould, Richard Hartley, and Dylan John Campbell. Deep declarative networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[29] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3):930-958, 2021.
[30] Alexander Meulemans, Matilde Tristany Farinha, Maria R. Cervera, João Sacramento, and Benjamin F. Grewe. Minimizing control for credit assignment with strong feedback. arXiv preprint arXiv:2204.07249, 2022.
[31] Paul Werbos. Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University, 1974.
[32] David E. Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning representations by back-propagating errors. Nature, 323(6083):533-536, 1986.
[33] Luís B. Almeida. Backpropagation in perceptrons with feedhack. In Rolf Eckmiller and Christoph v.d. Malsburg, editors, Neural Computers, pages 199-208. Springer Berlin Heidelberg, 1989.
[34] Fernando J. Pineda. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Computation, 1(2):611-172 1989.
[35] Nicolas Zucchet, Simon Schug, Johannes von Oswald, Dominic Zhao, and João Sacramento. A contrastive rule for meta-learming. Advances in Neural Information Processing Systems, 2022.
[36] James C. R. Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation, 2965): 2017.
[37] Karl Friston. The free-energy principle: rough guide to the brain? Trends in Cognitive Sciences, 1377):293-301, 2009.
12

---

[38] Miguel Carreira-
Carreira-Perpinan Perpinan and Weiran Wang. Distributed optimization of deeply nested systems. In Artificial Intelligence and Statistics, 2014.
[39] Dominik Dold, Akos F. Kungl, João Sacramento, Mihai A. Petrovici, Kaspar Schindler, Jonathan Binas, Yoshua Bengio, and Walter Senn. Lagrangian dynamics of dendritic microcireuits enables real-time backpropagation of errors. In Computational and Systems Neuroscience (Cosyne), 2019.
[40] Benjamin Scellier. A deep learning theory for neural networks grounded in physics. PhD Thesis, Université de Montréal, 2021.
[41] Jordan Guerguiev, Timothy P. Lillicrap, and Blake A. Richards. Towards deep learning with segregated dendrites. elife, 6:e22901, 2017.
[42] Mohamed Akrout, Collin Wilson, Peter Humphreys. Timothy P. Lillicrap, and Douglas B. Tweed. Deep learning without weight transport. In Advances in Neural Information Processing Systems, 2019.
[43] Alexander Meulemans, Francesco Carzaniga, Johan Suykens. João Sacramento, and Benjamin F. Grewe. A theoretical framework for target propagation. In Advances in Neural Information Processing Systems, 2020.
[44] William F. Podlaski and Christian K. Machens. Biological credit assignment through dynamic inversion of feedforward networks. Advances Neural Information Processing Systems, 2020.
[45] Isabella Pozzi, Sander Bohte, and Pieter Roelfsema. Attention-gated brain propagation: how the brain can implement reward- eeward-based error backpropagation. Advances in Neural Information Processing Systems, pages 2516-2526, 2020.
[46] Charles D. Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5):350-36)) -33550333,-10 2013.
[47] Satoshi Manita, Takayuki Suzuki, Chihiro Homma, Takashi Matsumoto, Maya Odagawa, Kazuyuki Yamada, Keisuke Ota, Chie Matsubara, Ayumu Inutsuka, Masaaki Sato, Masamichi Ohkura, Akihiro Yamanaka, Yuchio Yanagawa, Junichi Nakai, Yasunori Hayashi, Matthew E. Larkum, and Masanori Murayama. A top-down cortical circuit for accurate sensory perception. Neuron, 86(5): 1304-1366. 2015.
[48] Tiago Marques, Julia Nguyen, Gabriela Fioreze, and Leopoldo Petreanu. The functional organization of cortical feedback inputs to primary visual cortex. Nature Neuroscience, 21(5):
757-764, 2018
[49] Lisa Kirchberger, Sreedeep Mukherjee, Ulf H. Schnabel, Enny H. van Beest, Areg Barsegyan, Christiaan N. Levelt, J. Alexander Heimel, Jeannette A. M. Lorteije, Chris van der Togt, Matthew W. Self, and Pieter R. Roelfsema. The essential role of recurrent processing for figure-ground perception mice. Science Advances, eeabe 1833, 2021.
[50] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances Neurul Information Processing Systems, 2016.
[51] Alexander Meulemans, Matilde Tristany Farinha, Javier Garcia Ordonez, Pau Vil- imelis Aceituno, João Sacramento, and Benjamin Grewe. Credit assignment in neural networks through deep feedback control. In Advances in Neural Information Processing Systems,209 2021.
[52] Yann LeCun. The MNIST database of handwritten digits. Available at http://yann. lecun. comfexdb/nnist, 1998.
[53] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E. Hinton, and Timo- thy Lillicrap. Assessing the scalability of moologially-motivaae deep learning algorithms and architectures. In Advances in neural information processing systems, 2018.
13

---

[54] John F. Kolen and Jordan B. Pollack. Back-propagaan without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), 1994.
[55] Fabian A. Mikulasch, Lucas Rudelt, Michael Wibral, and Viola Priesemann. Dendritic predictive coding: A theory of cortical computation with spiking neurons. arXiv preprint arXiv:2205.053333 2022.
[56] Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, 1999.
[57] Andre M. Bastos, W. Martin Usrey, Rick A. Adams, George R. Mangun, Pascal Fries, and Karl J. Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695-711, 2012.
[58] Georg B. Keller and Thomas D. Mrsic-Flogel. Predictive processing: canonical cortical computation. Neuron, 100(2):424-435, 2018.
[59] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Multiplicative filter networks. September 2020.
[60] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[61] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016.
[62] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, 2018.
[63] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, 2020.
[64] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer US, 1998.
[65] Timothy E. J. Behrens, Timothy H Muller, James C. R. Whittington, Shirley Mark, Alon Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron, 100(():490-55)) 509,005090000 2018.
[66] Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 21 21(6):860-868, 2018.
[67] Jane X. Wang. Meta-learning in natural and artificial intelligence. Current Opinion Behavioral Sciences, 38:90-95, 2021.
[68] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, 2019.
[69] Stefano Fusi, Patrick J. Drew, and Larry F. Abbott. Cascade models of synaptically stored memories. Neuron, 45(4):599-611, 2005.
[70] Lorric Ziegler, Friedemann Zenke, David B. Kastner, and Wulfram Gerstner. Synaptic consol- idation: from synapses to behavioral modeling. Journal of Neuroscience, 35(3):1319-1333 1334, 2015.
[71] Marcus K. Benna and Stefano Fusi. Computational principles of synaptic memory consolidation. Nature Neuroscience, 19(12): 2016.
[72] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
14

---

[73] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis
drei Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114(13):3521-352 2017.
[74] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning. 2017.
[75] Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient- based tuning of continuous regularization hyperparameters. In International Conference on Machine Learning, 2016.
[76] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint 1 2003,0999, 2018.
[77] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference Artificial Intelligence and Statistics,
2020.
[78] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human- level concept learning through probabilistic program induction. Science, 133066666:13322110 13388, 2015.
[79] A. E. Bryson and Y. C. Ho. Applied optimal control: optimization, estimation, and control. Blaisdell Pub. Co., 1969.
[80] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43, 2018
[81] Yann LeCun. A theoretical framework for back-propagation. In Proceedings of the 1998 Connectionist Models Summer School, 1988.
[82] Aditya Gilra and Wulfram Gerstner. Predicting non-linear dynamics by stable local learning in recurrent spiking neural network. eLife, 6:e28295, 2017.
[83] Sophie Denève, Alireza Alemi, and Ralph Bourdoukan. The brain as efficient and robust adaptive learner. Neuron, 94(5):969-977, 2017.
[84] Alireza Alemi, Christian Machens, Sophie Deneve, and Jean-Jacques Slotine. Learning nonlin- ear dynamics in efficient, balanced spiking networks using local plasticity rules. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[85] Dmitri A. Rusakov, Leonid P. Savtchenko, and Peter E. Latham. Noisy synaptic conductance: Bug or a feature? Trends in Neurosciences, 43(6):363-3722
2020. 2020.
[86] J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, (110:33088-300 1984.
[87] Michael A. Cohen and Stephen Grossberg. Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, SMM-11355:815 826, 1983.
[88] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
15


Reviews:
[{"review_id": "kVQuAr8tjOk", "reviewer": "Reviewer_5Kjg", "summary": "This work proposes a theoretical framework for optimizing equilibrium neural networks by reformulating the original constrained optimization problem into a least-control problem. The least-control problem is further solved with a two-steps iterative procedure that first drives the networks dynamics with a control signal towards minimizing fixed points verifying the initial problem constraints, then taking a gradient step to minimize the norm of the control terminal cost. \n\nMotivations for such a principle are manifold: First, solutions are shown to coincide with the initial optimization problem under feasible conditions on the control that leave flexibility for its explicit construction. Second, credit assignement and activation dynamics are originally combined in a single dynamic that result in a spatially and temporally local explicit weight update rule. Third, this formulation can also be linked to models of free energy minimization, offering an original perspective on the learning objective operated by such models as a relaxation of the proposed least-control objective.\n\nSeveral experiment are conducted to showcase the learnability of such system for supervised image classification, implicit neural representation and meta-learning. The technique is shown to have comparative performance to the current leading technique for training equilibrium model recurrent backpropagation (RBP) and can encompass meta-learning with good results compared to gradient-based methods.", "questions": "I have three questions, with one regarding the neuroscientific interpretation of the update rule, a second for optimization and a third for the experiments:\n\n- In line 98 is stated that \"the resulting weight update now represents a local Hebbian rule multiplying presynaptic input with the postsynaptic control signal.\" Could you elaborate on this? Form equation (4), the weight update consist in the Jacobian of f at equilibirum multiplying the instantaneous variation of activity driven by f. Hence, the interpretation given by the authors is not straightfoward and does not seem to be local. Could you help clarify this point?\n\n - Could you relax the assumption of convexity of the function $L$ in proposition 3 to accomodate for locally convex functions? For instance, the meta-learning problem proposed in section 3.2 does not consider a convex loss function with respect to model parameters and the conditions for proposition 3/4 do not hold.\n\n - Regarding the experiments, can you provide more details regarding the computation cost of training such a system for the considered problems? It seems to me that the fixed point scheme for finding the equilibrium of the controlled dynamics would require careful tuning for finding a trade-off between rapidity and precision of the control estimate, that is not discussed in the empirical section. A temporal comparison with RBP would be welcome to characterize the efficiency of the optimization process.  \n\n", "limitations": "The work does not raise particular ethical or societal question. Conversely, it constitute an interesting direction of work for optimization for deep learning. The framework is however restricted to deep equilibrium model and an important development would be to extend such principle to arbitrary dynamics. Another interesting aspect of this work is the connection drawn with neuroscience-inspired models (predictive-coding and free-energy) that calls for more in-depth comparison. \n\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First, I find this works is nicely written, with a clear and rational development which makes it highly accessible. The technical problem formulation as well as the different connection with previous models are well executed and contributes to appreciating the original contribution of this work. The experiments are diverse and show the general applicability of the method.\n\nI would only note that a lot of the paper technicality is deferred to the supplementary, and while the mathematical development is clear, on the other hand, much of the choices for solving experimentally the least-control problems are not discussed in the main text. I understand that this is a choice for exhaustivity but I find that it undermines slightly the experimental section for demonstrating the value of the least-control formulation. Globally however, I have a very positive opinion on this work as it offers a valuable theoretical contribution to the field. \n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HkcqRM1QDrj", "reviewer": "Reviewer_GGtZ", "summary": "This paper presents a principled generalization of “Minimizing Control for Credit Assignment with Strong Feedback” (Meulemans et al, 2022) rooted in control theory and applying to any implicit model whose forward pass is defined as an equilibrium state. Following the same route as the aforementioned paper, the proposed approach builds upon the idea that credit assignment can be achieved by minimizing the amount of control needed to reach an equilibrium which minimizes the loss of interest (Eq. 1). This is referred to as the “least control principle” (LCP).\n\n- Subsection 2.1 is dedicated to how to train the recurrent feedforward weights. More precisely, the optimization problem at hand is defined (Eq. 2), as well as the subsequent notion of “optimal control” (Eq. 3). Theorem 1 shows that, assuming such an optimal control, the gradient of the amount of control reads as a relatively simple local learning rule.\n\n- Subsection 2.2 describes how to train the feedback weights. Namely, the control is specified as the linear mapping of a low-pass filtered loss gradient (with respect to the output units) onto each neuron (Eq. 6). Theorem 2 states that the resulting steady state control is optimal if it satisfies a “column space” (CS) condition (which can be remotely regarded as a kind of “weight symmetry” requirement) and in the limit of infinitely slow leaky integral controller ($\\alpha \\to 0$). The authors consequently prescribe general dynamics for the controller such that, by construction, the steady state control satisfies the CS condition (Eq.8). These dynamics are referred to as “dynamic inversion” (DI).\n\n- In Subsection 2.3, it is shown under which hypothesis least control solutions lead to solutions of the original learning problem (Eq. 1) in Prop. 4. Extra theoretical guarantees as to when the hypothesis of the theory do not hold exactly are provided in the Appendix. \n\\item In Subsection 2.4, the least control principle is re-casted into a constrained energy minimization problem, and the connection between this reformulation and predictive coding when the implicit function F is implemented by a feedforward network is discussed in the Appendix. This leads to the definition of energy-based control dynamics (Eq. 10).\n\n- In Subsection 3.1, the equilibrium RNN model is introduced (Eq. 11) along with the LCP feedforward weight updates. Different feedback circuitries to implement the control are proposed.\n\n- Subsubsection 3.1.1 proposes a control implemented as a direct linear feedback (LCP-LF). The rule used to train the feedback weights to achieve the CS condition is introduced in Meulemans et al (2022). Subsubsection 3.1.2. proposes one circuitry implementing dynamic inversion (LCP-DI, Eq. 8), another one implementing energy minimization, with only the former being experimentally tested. A more bioplausible implementation of DI with distinct feedback weights learned by Kollen-Pollack (LCP-DI, KP) learning rule is also considered. The resulting three algorithms (LCP-LF, LCP-DI, LCP-DI-KP) are tested on MNIST and achieve comparable accuracy with recurrent backpropagation (RBP). Next, LCP-DI is applied to a more complex deep equilibrium model (described in the Appendix) on CIFAR-10 and implicit neural representation (INR), with again an accuracy comparable to RBP.\n\n- In subsection 3.2, it is shown that LCP can be used as a single phase first-order metalearning technique. The resulting inner-loop parameter updates, as well as the outer-loop metaparameter updates read in a simple fashion and achieve superior performance to other first-order single phase metalearning techniques.\n\n- A thick appendix ($\\approx 40$ pages) is provided to support the main mathematical claims, as well as many extra mathematical results.", "questions": "Below, I shall refer to your approach as LCP.\n- L. 43: I think it’s a bit misleading to say that SDFC deals with “feedforward” architectures while LCP might extend the idea to “equilibrium architectures”: the architectures are also recurrent and converge to equilibrium in this past work! Wouldn’t it be a better way to phrase the difference in this regard (rather than feedforward models VS equilibrium models) that LCP can deal with any recurrent connectivity pattern?\n \n- L.93-99 (main): what is the difference between LCP applied to a implicit function with a feedforward connectivity pattern on the one hand, and SDFC on the other hand? My take is that LCP allows you to have Q to depend on $\\theta$ and $\\phi$ (as you point out in the appendix), and therefore use the general-purpose algorithm of Eq. 8, while you couldn’t in the SDFC setting since Q is assumed to be constant, but I’m not sure. It would help to have an extra section in the appendix comparing precisely SDFC and LCP in the case of a feedforward connectivity pattern.\n\n- The following point is particularly important. My understanding of the proposed approach is that the control applies to every single neuron of the dynamical system. If this is correct, in some sense, each neuron of the dynamical system receives a direct feedback from the controller – in this regard, it would be no surprise that the learning rule is local in space. However, I fail to see clearly if your learning theory extends to the case of partial control, e.g. only a subset of the neurons of the system are controlled by $\\psi$. For instance, imagine an implicit model defined as the steady state of a layered Hopfield network – this falls in the general category of implicit functions you are considering in this paper. Now imagine that only the last layer of the model is controlled. While the remaining neurons are not explicitly acted upon by the controller, they might well be influenced, through recurrent feedback connections, by the output neurons being controlled. The resulting displacement of the neurons might contain some error information (at the expense of time locality though), e.g.\n$\\hat{\\nabla}_{\\Phi} \\ell \\approx \\Phi_{controlled} - \\Phi_{not controlled}$.\nHowever, your learning rule prescribes a zero weight update for such neurons. What is happening here? I could have missed something here. But if not, I think it should be clearly stated that your theory does not cover this case and that the theory can only handle *direct* control. \n\n- The constrained energy minimization view is very interesting, as well as the discussion about the two proposed models in Eqs. (14) and (15). However you say “we do not expect major differences in practice between the two, as they have the same equilibrium”. Still, it would have been interesting to see the results you obtain, for instance on MNIST, with the energy-based approach to motivate it beyond the theoretical connection. It seems that you have gone only halfway through the energy-based approach. \n\n- I am a bit worried about the CIFAR-10 experiments. L. 1281, you say that the model is made up of an encoder, an implicit module and a decoder. How do you train the encoder $g(x,\\theta)$? The encoder is made up of conv layer and batchnorm layer. Unless you: 1/ explicitely augment the state \\phi of your model dynamics with the output of the conv layer and that of the BN (the same strategy that you apply to train the decoder) 2/ apply control $\\psi$ to these two, I cannot see how you proceed. Also, to run the controller dynamics (Eq. 147), you need to compute the Jacobians $\\partial f/\\partial \\phi$ and $\\partial h/\\partial\\phi$. However f reads as the complex composition of several convolutional and group normalization operations, and g as the composition of batchnorm, average pooling and linear operations. Don’t you need to backprop through f and g and therefore all these complex operations (batch norm, group normalization) to run the controller dynamics? If it is so, it should be clearly stated in the limitations of the paper. Beyond this, the batchnorm operation is not very bioplausible and other more simple (non parametric) normalization schemes have been proposed (like in this paper: https://arxiv.org/abs/2003.01513). If my understanding of the CIFAR-10 experiment is not correct, please provide more details in the Appendix. \n", "limitations": "My overall understanding of this paper is that credit assignment through control minimization (as introduced in Strong Deep Feedback Control) can be casted into an principled control theory-based approach, applying to any implicit model (as opposed to SDFC which only applies to converging RNNs with a feedforward connectivity pattern), which allows for $\\phi$ and $\\theta$ dependent feedback control weights $Q(\\theta, \\phi)$. Crucially, this allows to use a novel algorithm to satisfy the column space condition beyond the linear constant feedback matrix $Q$, possibly for more complex models on more complex tasks, as motivated in the main. However, as per my understanding, this new algorithm comes at the expense of having to backprop through the implicit model f and the decoder h, as it seems to be the case in the CIFAR-10 experiments. It's a little bit frustrating because the precise motivation of this new algorithm to train the feedback controller is to train more complex models for which the linear feedback controller might not sufficient for, and while you take care of not using BP through Kollen Pollack kind of mechanisms for the MNIST experiments, it seems to be overlooked in the CIFAR-10 experiments. Moreover, unless I misunderstood something again, it seems to me that the scope of the theory is limited to the case of a *direct* feedback upon *all* the neurons of the dynamical system. If all these remarks are sound -- which again may not be the case if I misunderstood something -- if should be very clearly stated in the limitations section. ", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written.\n- The theory is solid and all the mathematical content of the paper is accurate. I spent hours checking every single proof of the Appendix.\n- An experimental effort has been made to demonstrate that LCP applies to a large body of models and problems (MNIST, CIFAR-10, INR, Omniglot).\n- The proposed theory achieves credit assignment in a single phase – the forward pass and the gradient computation occur simultaneously. Such theories are rare in this literature (Payeur et al 2020, Sacramento et al 2018, Meulemans et al 2022). \n- The Appendix is extremely detailed and rigorous. \n\nWeaknesses:\n- 10 lines, deep in the Appendix of around 40 pages, to compare your approach with the work *closest to yours* is not enough to my eyes. More precisely, the present paper does not build enough explicit comparison with Strong Deep Feedback Control (SDFC) – “Minimizing Control for Credit Assignment with Strong Feedback”, Meulemans et al, 2022 – which is the closest to theirs. The only explicit comparison appears in S6.3.3, L.1094-1103 of the Appendix. In the light of SDFC, the least control principle idea itself no longer appears as much of an original idea – see Eq. 5 of that paper. Also this approach also already achieves locality in space and time. While the authors re-build from scratch all the maths from first principles of constrained optimization in their appendix, I am very surprised they chose not to include as well the detailed theoretical foundations of this past work.\n- The scope of the theory is not clear to me. Most importantly, I am not sure whether the theory extends or not to the case where only a subset of the neurons of the dynamical system are controlled. I detail my concern below. However, I might be wrong and I might well change my opinion about this if the authors address this concern accordingly. \n\n- The details of the CIFAR-10 experiments are hidden in the Appendix. I am a bit skeptical about the model choices – I’m afraid there could be some hidden backprop here. I am also disappointed about the 80% performance – for instance, equilibrium propagation along with the Kollen Pollack algorithm can achieve almost 87% test accuracy when applied to a Hopfield network without weight symmetry (Laborieux et al, 2021). I’ll come back to this in details in the questions part.\n\n- Finally, as per the NeurIPS 2020 reviewing guideline (https://nips.cc/Conferences/2020/CallForPapers): \"*While theoretically grounded arguments are encouraged, it is counterproductive to add “decorative math” whose primary purpose is to make the submission look more substantial or even intimidating, without adding significant insight*\". While I do appreciate a lot the degree of details and rigor of the appendix, I think the authors could have better selected the most important information to put in the Appendix. Also, I'm wondering whether it can be expected that reviewers go through 40 pages of appendix when reviewing multiple papers over a short timeframe. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9PBFbXhWXYA", "reviewer": "Reviewer_PoYW", "summary": "This paper proposes use of the least-control principle which minimizes the amount of optimal control needed at equilibrium as a local alternative to backpropagation in recurrent neural networks. The authors establish conditions that should be met at equilibria, relate what is optimized back to the standard optimization problem, and relate their framework to constrained energy minimization. Finally, the approach is validated to get similar performance to backpropagation on a supervised learning settings based on MNIST, CIFAR-10, and a natural image dataset as well as in meta-learning experiments on Omniglot. ", "questions": "- Could you provide an intuitive explanation of what it means to minimize the \"amount of optimal control needed at equilibrium\"? (i.e. from Figure 1)\n- Can you provide a clear set of advantage and disadvantages of the proposed framework with respect to the recurrent backpropagation baseline? \n- Is it possible to create an experiment that could empirically highlight those advantages or is it beyond the scope of available computation? \n- Can you provide more clarity about the comparison between you approach and iMAML? I was wondering why you only included iMAML for comparison rather than normal MAML as well?", "limitations": "The authors can make the contribution clearer by having a section that really addresses a direct comparison of advantages and limitations with respect to standard recurrent backpropagation. As someone who is not an expert in this specific proposed approach, it remained very unclear to me reading through the paper what precisely the benefits would be. The authors seems to be motivated by biological plausibility or scalability as the paper does not claim that backpropagation is ever surpassed in the experiments. In light of this, it is also disappointing that potential benefits in terms of scalability could not be validated empirically. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The authors provide a solid theoretical foundation for their approach in Theorems 1 and 2. \n- I really appreciate sections 2.3 and 2.4, which relate back to the implications for the original learning problem and alternative interpretations of the least-control principle. \n- It is nice to see application in both supervised learning and meta-learning settings, but it would be preferable to consider a larger breadth of datasets and architectures to provide a stronger validation of the proposed meta-learning approach. \n\nWeaknesses: \n- Authors match but do not improve on backpropagation in terms of performance and the authors did not clearly highlight what the additional advantages of this framework would be presumably related to scalability and biological plausibility. \n- The title should be refined to highlight the focus on biological plausibility and recurrent neural networks. I believe I bid on this paper thinking it would probably relate to multi-agent RL and/or game theory in RL based on the title. You probably want to give readers a better idea about exactly what topics this paper will focus on in your title. \n\nWhile I think this paper is strong in a number of ways, I lean towards rejection at the moment because I feel that I am missing some basic understanding about the underlying motivation for exploring this direction. However, I look forward to getting answers to my questions from the authors and seeing what the other reviewers have to say. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9-LrnNVyjN", "reviewer": "Reviewer_6XDX", "summary": "The submission introduces 'the least control principle' a novel paradigm for training neural networks with local learning rules. The learning algorithm is intimately related to predictive coding and energy based models. The authors show that the learning algorithm is flexible by applying it to RNNs and meta-learning tasks. \n\n___ \n\nPost rebuttal update: The authors have addressed most of my concerns. I have accordingly modified my score.", "questions": "I will gladly increase my score if/when the authors address the weaknesses in the previous section.", "limitations": "n/a", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The submission is overall very interesting and well written. It brings new insight to a field under active investigation.\n* The empirical demonstrations showcase the flexibility of the algorithm.\n* The relationship to the energy based models and the alternative limit of beta -> infinity is very interesting. \n\n**Minor Weaknesses**\n* The evaluation of the algorithm on a well studied and popular benchmark is missing. The experiments on RNN and meta learning are great but because they are less studied examples, it is harder for the average reader to compare performance of the algorithm. Would be good for the authors to demonstrate that they can use their algorithm to train a deep net on a classic vision task like ImageNet (or even CIFAR10 with a feedforward network).\n* The discussion of the limitations of the work is very narrow and implies that the method has no limitations beyond 'out-of-equilibrium' networks and explicit parameter dependence. However, for example we know that a major limitation of predictive coding is its practical inefficiency because of its equilibration phase. Doesn't least-control also suffer from the same issue? An honest discussion of practical limitations and compute requirements is missing. The compute details of the experiments is also missing. (Hardware/compute time for the different methods etc.)\n* What are the consequences of alpha not being zero?\n* Related to above, overall empirical demonstrations of the fate of the theorems in regions that they do not apply is lacking. For example what happens to thm 2, prop 3, prop 4 when alpha is finite or f is not strongly convex as in (11). The experiments showing that learning take place are good but it would be great to empirically show that the theorems/propositions still approximately hold if their conditions are not fully satisfied. Prop. S8 somewhat addresses this but assumes that the approximate state and optimal state are close. Is there a guarantee that this is the case? Might be a good idea overall to show one or two empirical plots exploring these issues.\n\n\n**Exposition** \nThe following are suggestions to improve the paper. (in no particular order)\n* (Line 114) 'y are at a target value'. This probably only holds when the objective is supervised (e.g. regression/classification). Might be good to clarify this to avoid confusion.\n* Sometimes the presentation is a little confusing and requires multiple reads. For example in section 2.2, the paragraphs jump between describing eq. 6 and describing the generality of thm 2. I would suggest the authors reword this paragraph such that the reader does not have to go back and forth as many times. (Pehraps provide the theorem first and then give examples and expand on the specific example.) In general, one would expect the paragraph before or after a theorem or proposition to explain the theorem/proposition in more understandable terms.  \n* Another example of above is after Eq. 4. The authors alternate between describing their own algorithm and contrasting with backprop. I would suggest first describing eq. 4 in terms of Hebbian learning and then moving on to contrasting it with backprop. \n* Might be good to mention 'predictive coding' when referring to the model of Whittington and Bogacz (line 189). \n* Would be good to explicitly show the model parameter updates (eq. 4) for the specific examples of section 3. This would make it easier for the reader to see that they are indeed Hebbian. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept