Nature-Inspired Local Propagation
Authors: Alessandro Betti
Marco Gori
Venue: NeurIPS 2024 poster (2024)
Keywords: hamilton equation
backpropagation
Primary area: learning_theory

Abstract: The spectacular results achieved in machine learning, including the recent advances in generative AI, rely on large data collections. On the opposite, intelligent processes in nature arises without the need for such collections, but simply by on-line processing of the environmental information. In particular, natural learning processes rely on mechanisms where data representation and learning are intertwined in such a way to respect spatiotemporal locality. This paper shows that such a feature arises from a pre-algorithmic view of learning that is inspired by related studies in Theoretical Physics. We show that the algorithmic interpretation of the derived “laws of learning”, which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity. This opens the doors to machine learning studies based on full on-line information processing that are based on the replacement of Backpropagation with the proposed spatiotemporal local algorithm.

Nature-Inspired Local Propagation
Alessandro Betti
Marco Gori DISSM University of Siena Siena, Italy marco. gorco.gori@umisi..
IMT School for Advanced Studies Lucca, Italy
Abstract
The spectacular results achieved machine learning, including the recent advances in generative AI, rely large data collections. On the opposite, intelligent pro- cesses in nature arises without the need for such collections, but simply by on-line processing of the environmental information. In particular, natural learning pro- cesses rely on mechanisms where data representation and learning are intertwined in such way to respect spatiotemporal locality. This paper shows that such a feature arises from a pre-algorithmic view of learning that is inspired by related studies in Theoretical Physics. We show that the algorithmic interpretation of the derived "laws of learning", which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity. This opens the doors to machine learning studies based on full on-line information processing that are based on the replacement of Backpropagation with the proposed spatiotemporal local algorithm.
Introduction
By and large, the spectacular results of Machine Learning in nearly any application domain strongly rely on large data collections along with associated professional skills. Interestingly, the successful artificial schemes that we have been experimenting under this framework are far away from the solutions that Biology seems have discovered. We have recently seen a remarkable effort in the scientific community to explore biologically inspired models (e.g. see [1 crucial role of temporal information processing it is clearly identified.
16 30 18]
where the
While this paper is related to those investigations, it is based on more strict assumptions on environ- mental interactions that might stimulate efforts towards more radical transformation of machine learning with emphasis on the temporal domain. In particular, we assume that learning and inference develop jointly under nature based protocol of environmental interactions and then suggest developing computational learning schemes regardless of biological solutions. Basically, the agent not given the privilege recording the temporal stream, but only represent it properly by appropriate abstraction mechanisms. While the agent can obviously use its internal memory for storing those representations, we assume that it cannot access data collection. Instead, the agent can only rely on buffers of limited size to retain the information it acquires. From a cognitive perspective, these small buffers allow the agent review recent inputs backward in time, implementing form of selective attention.
We propose a pre-algorithmic framework which derives from the formulation of learning Optimal Control problem [99] and propose an approach to its solution that is also inspired by principles of Theoretical Physics. We formulate the continuous learning problem to emphasize how optimization theory brings out solutions based on differential equations that recall similar laws in nature. The discrete counterpart (2) p. 2], which more similar recurrent neural network algorithms that are found in the literature, can be derived as numerical method and applied in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

practical scenarios like lifelong learning with long video streams [4] where an Euler method for the differential equations can serve as an "optimizer" for RNN weights. Interestingly, we demonstrate that the online computation described in this paper achieves spatiotemporal locality, thereby contributing
to the longstanding debate on the biological plausibility of Backpropagation g 33 21]. Specifically, we address the update locking problem and the issue of infinitely fast signal propagation in neural networks. Finally, the paper shows that the conquest of locality opens up a fundamental problem, namely that of approximating the solution of Hamilton's 's equations with boundary conditions using only initial conditions. A few insights on the solution of this problem are given for the task of tracking in optimal control, which opens the doors of a massive investigation of the proposed approach.
2 Recurrent Neural Networks and spatiotemporal locality
We put ourselves in the general case where the computational model that we are considering is based on digraph D A) where V 21. n)
is the set of vertices and and A is the set of directed arches that defines the structure of the graph. Let ch(1) denote the set of vertices that are children of vertex and with pa(i) the set of vertices that are parents of vertex for any given i V. More precisely we are interested in the computation of neuron outputs over a temporal horizon [0, T). Formally, this involves assigning each vertex i V trajectory based on the outputs of other neurons and environmental information. The environmental information mathematically represented by trajectory u: [0, +00) + Rd, We will assume that the output of the first d neurons (i.e the value of x; for 1 = 1....,d d) matches the value of the components of the input: (t()) u,(t) for i = I.... d and Vi [0, T". In order to consistently interpret the first d neurons as input we require two additional property of the graph structure:
x:;(() of outputs that is computed
pa(i) = 0 Vi = ld... va(dd+l,........
(1)
dJ.
(2)
Here (T) says that an input neuron do not have parents, and it also implies that no self loops are allowed for the input neurons. On the other hand (2) means that all input neurons are connected to at least one other neuron amongst {d + I....,,! n).
We will denote with zt(t) (without a subscript) the ordered list of all the output of the neurons at time t except for the input neurons, x=t)= == ( Fn (f)), and with this definition we can represent xz) for any E jo, T] as vector in the euclidean space Ra-d, This vector is usually called the state of the network since its knowledge gives you the precise value of each neuron in the net. The parameters of the model are instead associated to the ares of the graph via the map (j. A + wa where Why assumes values on R. We will denote with wes (t) E Rl pa(t)) the vector composed of all the weights corresponding to arches of the form (j. If we let N == E |pa(i) the total number of weights of the model we also define R 3 un()= == (wi (l),... (()) the concatenation of all the weights of the network. Finally we will assume that the output of the model is computed in terms of a subset of the neurons. More precisely we will assume that, given a vector of m indices (4 .u.....,u (m) with i {d +
n), ..)...)) at each temporal instant the output of the net is function
T: RT > Rd of (Iay,.... (........ Fim)) ). For future convenience we will denote O (ii...., }.
general we are interested in computational schemes which are both local in time and causal. Let us assume that are working at some fixed temporal resolution meaning that we can define a partition of the half line (0, +oo), P == {0- = t < < ... < t < .... with tn = t fn-1 + T, then the input signal becomes sequence of vectors (U²) with U (((()) and the neural outputs and parameters can be regarded as an approximation of the trajectories and w: X² ss (()) and W ss w(()), ), n = 1, [T/T]. A local computational rule for the neural outputs means that X is a function of X -
Temporal locality and causality
X X W W W W and
In the reminder of the paper we will try whenever possible to formally introduce functions by elearly stating domain and co-domain. In particular whenever the function acts on product space we will try to use a consistent notation for the elements in the various sets that define the input so that we can re-use such notation to denote the partial derivative of such function. For instance let suppose that f: A B - R is function that maps (a, 6) + (a, b) for all a E and b E B. Then we will denote with fa the function that represents the partial derivative of f with respect to its first argument, with fo the partial derivative of f with respect to its second argument as function and so on. We will instead denote, for instance, with f. (z, y) the element of R that represent the value of fa on the point (x. y) E A x B.

---

tr-t, where [ (first order method) then
< T/r can be thought as the order of locality. If we assume that
X F(X;-),11 X X W WT
1 1, 17+1).
(3)
Causality instead expresses the fact that only past information can influence the current state of the variables meaning that actually (3) should be replaced by X; = F(X;-1, w 1,1-)) Returning to the continuous description, this equation can be interpreted as a discretization of a Cauchy problem for
á f(z, w, t),
with assigned initial conditions on (0). Note that the ability determine the solution by evolving the state from a specified initial value is fundamentally due to our causality requirement.
Spatial locality Furthermore assume that such computational scheme is local in time and make use only on spatially local (with respect to the structure of the graph) quantities as follows:
(t) (t) c =t()) It' (z,(tt), PA (z(t)), IN' (w))))
for i = 1....d d and Vt E [0, T):
(5)
for i + I,...,n and Vt [0,
Here o; > for all i d 1, ,,...,, n sets the velocity constant that controls the updates of the i-th neuron, wt: R x R pa(i) RR pg() + R for all i d + 1,.... performs the mapping (r, 8) + (r, a, 9) for all R, az B E R pa(i), PA': R" - Rl pa(()) project the vector e Rn-d + PA*(E) on the subspace generated by neurons which are in pa(i) and IN': RR - R pa(i) maps the any vector w E RR IN' (w) onto the space spanned by only the weights associated to ares that points to neuron i. The assumptions summarized above describe the basic properties of RNN or, as sometimes is referred to when dealing with a continuous time computation, a Continuous Time RNN 32) The typical form of function V is the following
V V'(r, a. 33 - aB a), Vr E and Va, B E R pa(O]
where in this case is the standard scalar product on R pa(t)] and σ: R + R is a nonlinear bounded smooth function (usually sigmoid-lit like activation function). Under this assumption the state equation in (5) becomes
a (t) - (t) (IN'(w(() (x))) = -x, +
Σ (1) jEpa(()
(7)
which is indeed the classical neural computation. Here sketch result on the Bounded Input Bounded Output (BIBO) stability of this class of recurrent neural network which is also important for the learning process that will be described later. Propostion 1. The recurrent neural network defined by ODE 7 is (BIBO) stable.
Proof. See Appendix D
3 Learning as a Variational Problem
In the computational model described in Section 2 once the graph D and an input u are assigned, the dynamics of the model is determined solely by the functions that describes the changes of the weights over time. Inspired by the Cognitive Action Principle 3 that formulate learning for FNN in terms of a variational problem, we claim that in an online setting the laws of learning for recurrent architectures can also be characterized by minimality of class of functional. what follows we will then consider variational problems for a functional of the form
F(w)= = L'
| + (t),
d(t) dt,
(8)
where 20., w) is the solution of (4) with fixed initial condition 3
Φ: [o, T + IR a strictly positive smooth function that weights the integrand, m > 0, L: RM x R x [o. T] R. is a positive function and finally := Ei cj/(n - d). We discuss the requirements for making the stationarity conditions of this class of functional both temporally and spatially local and how they can be interpreted learning rules.
We do not explicitly indicate the dependence the initial condition avoid cumbersome notation.
3

---

3.1 Optimal Control Approach
The problem of minimizing the functional in (8) can be solved by making use of the formalism of Optimal Control. The first step is to put this problem in the canonical form by introducing an additional control variable follow
G(u) - //-
mc |0|2 + cl(w((t v), =(t: t) @(t) dt,
(9)
2
where w((;v) and sr(1; v) solve
i(t) [(((t), w()),)), (), and i0() v(I)
(10)
Then, the minimality conditions can be expressed in terms of the Hamiltonian function (see Ap- pendix Ap. that is defined for every & E R., w E R", R E IR and E [0, '] as:
1 q² x(1) 2mn + cllw, E,(((1)))( +p w),,
(E, w, w.q.tt)
(11)
via the following general result.
Theorem (Hamilton equations). Let H be in
11 and assume that x²()) x and w(0) w
are given. Then a minimum of the functional in (9) satisfies the Hamilton equations:
dbtt) = ((x((),
it()) Pw Pwo(t)/mmooff
(12)
Pt(t)= (t) fe(x(t), ((),
(t) (t) fw w(t), clw (lu(
together with the boundary conditions
Pz( = Pw (T) = O.
(13)
Proof. See Appendix A
3.2 Recovering tempora locality
Starting from the general expressions for the stationarity conditions expressed by (12) and 13 will now discuss how the temporal and spatial locality assumptions that we made on our computational model Section 2 leads to spatial and temporal locality of the update rules of the parameters w.
Temporal Locality The local structure of (TO). that comes from the locality of the computational model that we discussed in Section 2 guarantees the locality of Hamilton's 's equations 12 However the functional in (9) has a global nature (it is an integral over the whole temporal interval) and the differential term m /2/ links the value of the parameters across near temporal instant giving rise to boundary conditions (3). This also means that, strictly speaking (12) and (13 overall define a problem that is non-local in time. We will devote the entire Section to discuss this central issue.
Spatial Locality The spatial locality of (12) directly comes from the specific form of the dynamical system in (5) and from set of assumptions on the form of the term é. In particular we have the following result:
= 5) + L(E, 8) for every Ew) E IRN x Ra-d x [o,T], where V: RR x J0, T] +1+R, RR,. is regularization term on the weight and L: R=-d [0, + R, depends only the subset of neurons from which assume the outpul of the model is computed, that is Lg. (E = (E.(5.)) where Lo is the indicator function of the set of the output neurons. Let T be as in for all = d 1,...,n,
Theorem 2. Let
n, then the generic Hamilton's equations described in 12
become
6 " -x; to ( Ejepa(o) ( myyy) da -p/d/ (me))
(14)
pi c.p' ExEch(i) Coo (ESepacR) Wky ph cLg. (a, t)o Pid (f) -C:Pio' (Emepati) Wimeim Ir ck ckV (w, t)o
a typical choice for this function could be (ww, 8) 1w/2/2 with &

---

Proof. See Appendix B
Remark Notice directly inherit the spatially local structure from the assumption in 5
Theorem 2 other than giving us spatio-temporal rules show that the computation of the Pr has very distinctive and familiar property: for each neuron the values of plz are computed using quantities defined on children's 's nodes as happens for the computations of the gradients the Backpropagation algorithm for FNN. In order to better understand Eq. (14) let us define an appropriately normalized costate
(t) d'(((()) 6(t)
(t),, with at
Σ mEpa(1)
WammEm
Vi = d + ,,...,,,, n, (15)
where we have introduced the notation (t) to stand for the activation of neuron definitions ready state the following result Propostion 2. The differential system in is equivalent to the following system of ODE of mixed orders:
With these
C/ 4; +o(a);;
Φ Wij + 6 NL,, + k Vals (w,t);
à
Kig =
me
m
(16)
A4 + dt
log (a;)) ) + X - o'()a)
Σ
cadbuni cLq. t)o)))(((a)
kEch(i)
where X₂ defined as
15
Proof. See Appendix C
This in an interesting result especially since via the following corollary gives a direct link between the rescaled costates As and the delta error of Backprop:
Corollary (Reduction to Backprop). Let os be the same for all i = ,...., n so that now e₁ = c,
then the formal limit of the ) equation in the system 16 as c + Do is
X = o'.aa)
Σ
Nuuss + Lg, (z, (a,).
(17)
kEch(()
Proof. Dividing both sides of the equation for X in Eq. 16 by we get:
X à d c
(ai)) X X d's)) Σ Xwua= (x, (z, (a.). kkch(1)
c
Φ
dt
As + oo, the terms proportional to 1/c vanish, leaving us exactly with Eq.
Notice that Eq. 17 is exactly the update equation for delta errors in backpropagation: when i is an output neuron the value of X directly given by the gradient of the error, otherwise is express as a sum on its children (see (I3).
4 From boundary to Cauchy's conditions
While discussing temporal locality in Section S
we came across the problem of the left boundary conditions on the costate variables. We already noticed that these constraints spoil the locality of the differential equations that describe the minimality conditions the variational problem at hand. In general, this prevents us from computing such solutions with forward/causal scheme.
The following examples should suffice to explain that, in general, this is crucial issue and should serve as motivation for the further investigation we propose in the present section.
'We have avoided to introduce the notation until now because we believe that it is worth writing (14 with the explicit dependence on the variable and x least once better appreciate its overall structure.
5

---

Example 1. Consider case in which E. (ww, s), i.e. i.e we want to study the minimization
problem for 0o' (m [tt) /2+ /2+ v), (t)dt under the constraint i v. Then the dynamical
equation (1) = [x(1), uo))) does not represent constraint on variational problems for functional in (9). If we look at the Hamilton equation for pa. in (12) this reduces to f0x
P2 sw- We for all E [o, T). Indeed this is the solution that we would find if we pair Ps = -p= fw with its boundary condition py (I") 0 in (33). Notice that in general without this condition random Cauchy initialization of this equation would not give null solution for the Pr. Now assume that o exp(Ot) with 0 > 0, and m 1. Assume, furthermord that V(w, s) /2/ The functional I (l + (2)e/ dt defined over the functional space ([O, T); R is coercive and lower-semicontimuuus, and hence admits a minimum (see ITI. Furthermore one can prove (see (20)) that such minimum is actually C ( T]; (R*). This allows us to describe such minumum with the Hamilton equations described in (12). In particular as we already commented the relevant equations are only that for i and Pw that (t())
would however expect Ps. (t)
(t)
and Pun (t) - we with Pw (T) = 0. This first order system of equations is equivalent to the second order differential equation (t) + (t)r(t) (t) = 0. Each component of this second order system will, in general have an unstable behaviour since one of the eigenvalues is always real and positive.
This is a strong indication that when solving Hamilton's 's equations with an initial condition on pw we
will end up with a solution that is far from the minimum.
In the next subsection, we will analyze this issue in more detail and present some alternative ideas that can be used to leverage Hamilton's equations for finding causal online solutions.
4.1 Time Reversal of the Costate
In Example [ we diseussed how the forward solution of Hamilton's (12) with initial conditions both on the state and on the costate in general cannot be related to any form of minimality of the cost function in (9) and this has do with the fact that the proper minima are characterized also by left boundary conditions 13 The final conditions on Pa and Pw suggest that the costate equations should be solved backward in time, Starting from the final temporal horizon and going backward in time is also the idea behind dynamic programming, which is of the main ideas at the very core of optimal control theory.
Autonomous systems of ODE with terminal boundary conditions can be solved "backwards" by time reversal [9] p. 597] operation -/ and transforming terminal into initial conditions. More precisely the following classical result holds:
Propostion 3. Let (s) = (t)( be a system of ODEs on [0, T] with terminal conditions T(T) = yT and let p be the time reversal transformation maps t + T then j(s) v (s) ytt)
satisfies ((s) (s))s)) with initial condition g()) y²
Clearly 12 or 16 are not autonomous system and hence we cannot apply directly Proposition] 3 nonetheless, we can still consider the following modification of [14
C 3; = II; + a Ejepa(() wijaz) dbs (p///moo)
p -c,p'_ + ExEch(o) Goo (Espacce) WKyzt ) Pr + che, (2, t)o
(18)
(t) = c.pito' (Emepatt)) Wmmmm Ja,, + ckV (w, t)Φ
which are obtained from 14 by changing the sign to pa and pw: Recalling the definition of the rescaled costates (15) can cast, in the same spirit of Proposition 2 a system of equations without Pw- In particular we have as a corollary of Proposition 2 that
'The same argument that we give in this example works for a larger class of cocrcive potentials V. 'These are called Sobolev spaces, for more details see [

---

Corollary 2. The ODE system in 18 is equivalent to
C/3 t; = -- I; + (a:);
c Nu
h Vais (w, ();
Φ Whej
mc
m
(19)
d log(o' (a:)) - as
) N + (ass)
1 +
Σ kEch(()
Ca + cle,, (a,, 1) (as),
Φ
di
Proof. Let consider (16). The change of sign of pvo only affect the signs of Xgary and Vas, (we, in the WH equation, while the change of sign of p. result in a sign change of the term c. o' da(a)) Ekeeb(() C. and Lg, (z. (as) in the equation for Xς.
Equation i9 indeed particularly interesting because it offers an interpretation the dynamics of the weights that is in the spirit of a gradient-base optimization method. In particular this allow us the extend the result that we gave in Corollary to a full statement on the resulting optimization method:
Propostion 4 (GD with momentum). Let os be the same for all i 1,..., , n so that now O2 c, and let Φ(t) exp( with 0 > 0 then the formal limit of the system in 19 as - 00 is
x2 oai);
ū.g = -büig no N1 (k/m) (w, t);
(20)
A = ('(a)) Exech(H) Veusi + Le, (I. ()a()))))
Remark 2. This result shows that least in the case of infinite speed of propagation of the signal across the network (c - oc) the dynamics of the weights prescribed by Hamilton's 's equation with the costate dynamics that is reversed (the sign of P. and Pw is changed) results in gradient flow dynamic (heavy-ball dynamics) that it is interpretable as a gradient descent with momentum in the discrete. This is true since the term Na in this limit is exactly the Backprop factorization of the gradient of the term L with respect to the weights.
In view of this remark we can therefore conjecture that also for e fixed: Conjecture 1. Equation 79 is a local optimization scheme for the loss term E.
Such result would enable us to use 19 with initial Cauchy conditions as desired.
4.2 Continuous Time Reversal of State and Costate
Now we show that another possible approach to the problem of solving Hamilton's 's equation with Cauchy's 's conditions to perform simultaneous time-reversal of both state and costate equation. Since in this case the sign flip involves both the Hamiltonian equations the approach is referred to as Hamiltonian Sign Flip (HSF). In order to introduce the idea let us begin with the following example. Example 2 (LQ control). Let us consider a linear quadratic scalar problem where the functional in (9) is G(v) fo" qx² /2 + ro? /2//2222 dt and å ar + bu with q. positive and and b real parameters. The associated Hamilton's 's equations in this case are
sp, 1 = q ap; where s = =b2/rr. These equation can be solved with the ansatz p(t) = (t))((t), where @ is some unknown parameter. Differentiating this expression with respect to time we obtain
i
(21)
Ó -= (p 02)/2x
(22)
and using the 21 into this expression we find Í sB2 2a0 - -= = 0 which is known as Riccati
equation. and since p(T) = O, because of boundary (13) this implies ((T) = 0. Again if instead
we try to solve this equation with initial condition we end up with an unstable solution. However
0 solves an autonomous ODE with final condition, hence by Proposition 3 we can solve it with 0
initial conditions as long as we change the sign of é. Indeed the equation 0 + s02 + 2a0 + 0 is asymptotically stable and returns the correct solution the Riccati algebraic equation. Now the
crucial observation is that, as we can see from (22) ) the sign flip of Ó is equivalent to the simultaneous sign flip of E and p.
7

---

In Example 2 as we observe from (22). the sign flip of & is equivalent to the simultaneous sign flip of i and p. Inspired by the fact, let us associate the general Hamilton's 's equation ((2).. to this system the Cauchy problem
i(t)
f(x(t), t()),
(t)
Pw P=x(t) ((), clg (w(t), (tt)
(t)
= x(t)
-Pa(t) P=z
(23)
Pw (t)
where for all & [0, TI, s(1) {0, 13. Here we propose two different strategies that extends the sign flip discussed for the LQ problem.
Hamiltonian Track The basic idea is enforce system stabilization by choosing s(t) to bound both the Hamiltonian variables. This leads to define a Hamiltonian track:
Definition 1. Let S(E, w. P: q) (R"-dx x ( for every (E, p,, q) (RR--xx x RRy 2 be bounded connected set and let X (t) any continuous trajectory in the space (Ra-d d x R2 then we refer to
(()()(X(t)) t [0, [o. T) x (R*-d x R2
as Hamiltonian track (HT).
Then we define st) as follow
s(t) = {L,
if otherwise
(t), E S((x((), w(t), Pa pa()),
(24)
For instance if we choose S(C, = {(S, p. l + w// + |p| + gl| < R} we are constraining the dynamics of (23) to be bounded since each time the trajectory H (zr(L), w((), Pr. (())))(()) moves outside of a ball of radius R we are reversing the dynamics by enforeing stability.
Hamiltonian Sign Flip Strategy and time reversal We can easily see that the sign flip driven by the policy of enforcing the system dynamics into the HT corresponds with time reversal of the trajectory, which can nicely be interpreted as focus of attention mechanism. A simple approximation of the movement into the HT that of selecting s(t) sign(cos( (w)ss) where appropriate flipping frequency which governs the movement into the HT. In the discrete setting of computation the strategy consists of flipping the right-side of Hamiltonian equations sign with a given period. In the extreme case the sign flip takes place at any Fuler discretization step. Here we report the application of the Hamiltonian Sign Flip strategy to the classic Linear Quadratic Tracking (LQQT) problem by using recurrent neural network based reported experiments is to validate the HSF policy, which is in fact of crucial importance in order to exploit the power of the local propagation presented in the paper, since the proposed policy enables on-line processing.
2 f is an
fully-connected digraph. The purpose of the
The pre-algori framework proposed in the paper, which is based on ODE can promptly give rise to algorithmic interpretations by numerical solutions. the reported experiments we used Euler's discretization (see Appendix F for both architectural and algorithmic details).
Sinusoidal signals: The effect of the accuracy parameter. In this experiment we used sinusoidal target and a recurrent neural network with five neurons, while the objective function was G(v) fo q(20 =)2 /2 r|ul2//2 /2 + r'e [2|2 dl, where we also introduced a regularization term the weights. Here, xo denotes the neuron designated as the output (see Appendix E and r and r., are positive parameters. The HSF policy gives rise to the expected approximation results. In Fig. we can appreciate the effect of the increment of the accuracy term.
Tracking under hard predictability conditions. This experiment was conceived to assess the capa- bilities of the same small recurrent neural network with five neurons to track a signal which was purposely generated to be quite hard to predict. It is composed of patching intervals with cosine functions with constants.
The experimental analysis on this and related examples confirms effectiveness of the HSF policy shown in Fig. 3 Figure shows the behavior of the Lagrangian and of the Hamiltonian term, with the latter providing insights into the energy exchange with the environment.
8

---

1000
2000
Figure 1: Recurrent net with 5 neurons, q 10 (accuracy term), Tw (weight regularization term),
r 0.1 (derivative of the weight term).
1000
2000
Figure 2: Recurrent net with 5 neurons, g = 1000 (accuracy term), ri (weight regularization
term), 0.1 (derivative of the weight term).
5 Related Work
Optimal control. Optimal control theory primarily studies minimality problems for dynamical systems 6 The main complementary approaches to the problem the Pontryagin Maximum Principle [0] and dynamic programming. Additionally, as general minimization problem, both approaches significantly intersect with the calculus of variations 12
Optimal control for discrete
problems is also a classic topic [2] 2].
Neural ODE. Recent works, such as and subsequent studies 11 24 have applied results from optimal control to develop learning algorithms based on differential equations. However, these approaches differ significantly from the continual online learning considered in this work, as the time variable in the class of ODEs they examine is not tied to the input signal that represents the flow of the learning environment.
Online. On the other hand several works propose formulate the learning problems online and from a single stream of data [22] 34]. The classical approach to learn RNNs online is RTRL [15]; several approaches has been since proposed to reduce the high space/time complexities due the progressive update of a Jacobian matrix (23). In our method no storing of Jacobian matrices happens, hence the proposed method not senernlivation//efommaaaaioo of RTRL not related approaches
like 35
Nature-inspired computations. The primary distinction of our approach in discussing the biologi- cal plausibility of backpropagation lies in our development of a theory grounded entirely in temporal analysis within the environment and the concept of learning over time. While several classical (8 and recent approaches 29
27 14 share certain locality properties outlined here, they are primarily inspired by brain physiology. Similarly, most works that examine the biological plausibility
25
26
of backpropagation 8 33 21
overlook the role of time in the sense that we present in this work. Here, we propose laws neural propagation where connections are updated progressively over time, mirroring processes observed in nature.
Conclusions
This paper is motivated by the idea of proposing learning scheme that, like in nature, arises without needing data collections, but simply by on-line processing paper gives two main contributions. First, it introduces a local spatiotemporal pre-algorithmic framework that is inspired to classic Hamiltonian equations. is shown that the corresponding algorithmic formalization leads to the interpretation of Backpropagation limit case of the
environmental interactions. The

---

WMMMAMMN
1000
2000
Figure 3: Tracking highly-unpredictable signal: number of neurons: 5, q 100 (accuracy), weight
reg = 1. derivative of weight reg 0.1.
H
200
100
0
-000
0
1000
2000
Figure 4: Evolution of the Lagrangian and of the Hamiltonian function for the experiment whose tracking is shown in Fig.
proposed diffusion process in case of infinite velocity. This sheds light on the longstanding discussion on the biological plausibility of Backpropagation, since the proposed computational scheme is local in both space and time. This strong result indissolubly intertwined with a strong limitation. The theory enables such locality under the assumption that the associated ordinary differential equations are solved as a boundary problem. The second result of the paper is that of proposing method for approximating the solution of the Hamiltonian problem with boundary conditions by using Cauchy's initial conditions. In particular we show that we can stabilize the learning process by appropriate schemes of time reversal that are related to focus of attention mechanisms. We provide experimental evidence the effect of the proposed Hamiltonian Sign Flip policy for problems of tracking in automatic control. While the proposed local propagation scheme is optimal in the temporal setting classic related learning algorithms like BPTT and RTRL, the given results show that there is free lunch: The distinguishing feature of spatiotemporal locality needs to be sustained by appropriate movement policies into the Hamiltonian Track. We expect that other solutions better than the HSF policy herein proposed can be developed when dealing with real-word problems and may offer potential approaches to classic challenges in lifelong learning, such forgetting, that remain open and are not fully addressed by the current framework. This paper must only be regarded as theoretical contribution which offers a new pre-algorithmic view of neural propagation. While the provided experiments support the theory, the application to real- world problems need to activate substantial joint research efforts on different application domains.
and overcomes the limitations
Acknowledgments
We thank Stefano Melacci and Giovanni Bellettini for insightful discussions.
References
[1] Martino Bardi, Italo Capuzzo Dolcetta, et al. Optimal control and viscosity solutions of Hamiton-Jacabi-Bellln Bellman equations, volume 12. Springer, 1997.
[2] Dimitri Bertsekas. Abstract dynamic programming. Athena Scientific, 2022.
[3] Alessandro Betti, Marco Gori, and Stefano Melacci. Cognitive action laws: The case of visual features. IEEE transactions on neural networks and learning systems, 31(3):938-949 2019.
[4] Alessandro Betti, Marco Gori, and Stefano Melacci. Learning visual features under motion invariance. Neural Networks, 126:275-299, 2020.
[5] Haim Brezis and Haim Brézis. Functional analysis, Sobolev spaces and partial differential equations, volume 2. Springer, 201
10

---

[6] Piermarco Cannarsa and Carlo Sinestrari. Semiconcave functions, HamiltonJacobi equations, and optimal control, volume 58. Springer Science & Business Media, 2004.
[7] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[8] Francis Crick. The recent excitement about neural networks. N (TT 1989.
[9] Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Society, 2010.
[10] RV Gamkrelidze, Lev Semenovich Pontrjagin, and Vladimir Grigor'evic Boltjanskij. The mathematical theory of optimal processes. Macmillan Company, 1964.
[11] Mariano Giaquinta and Stefan Hildebrandt. Calculus of variations i. Calculus of Variations,
1:1, 1995.
[12] Mariano Giaquinta and Stefan Hildebrandt. Calculus of variations II, volume 311. Springer Science & Business Media, 2013.
[13] Marco Gori, Alessandro Betti, and Stefano Melacci. Machine Learning: A constraint-based approach. Elsevier, 2023.
[14] Geoffrey Hinton.
The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.133453 2022.
[15] Kazuki Irie, Anand Gopalakrishnan, and Jürgen Schmidhuber. Exploring the promise and limits of real-time recurrent learning, arXiv preprint arXiv:2305.19044 2023.
[16] Jack Kendall. A gradient estimator for time-varying electrical networks with non non-linear dissipa- tion. arXiv preprint arXiv:2103.05636 2021.
[17] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[18] Axel Laborieux and Friedemann Zenke. Holomorphic equilibrium propagation computes exact gradients through finite size oscillations. Advances in Neural Information Processing Systems, 15:100501 2022.
[19] Yann LeCun, D Touresky, Hinton, and Sejnowski. A theoretical framework for back- propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21-28, 1988.
[20] Matthias Licro and Ulisse Stefanelli. new minimum principle for lagrangian mechanics. Journal of nonlinear science, 23:179-204: 2013.
[21] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
[22] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
[23] Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. Journal of machine learning research, 21(135):1-34, 2020.
[24] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dis- secting neural odes. Advances Neural Information Processing Systems, 33:3952-3963, 2020.
[25] Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Predictive coding approxi- mates backprop along arbitrary computation graphs. Neural Computation, 34(6):1329-133 1368, 2022.
11

---

[26] Alexander Ororbia and Ankur Mali. The predictive barward-fowwit algorithm. arXiv preprint arXiv:2301.01452, 2023.
[27] Alexander Ororbia and Ankur Mali. Biologically motivated algorithms for propagating local target representations. In Proceedings of the aaai conference arlificial intelligence, volume 33, pages 4651 4658, 2019.
[28] Rao. Predictive coding in the visual cortex. Nature Neuroscience, (1),9-10, 1999.
[29] Tommaso Salvatori, Ankur Mali, Christopher I. Buckley, Thomas Lukasiewicz, Rajesh PN Rao, Karl Friston, and Alexander Ororbia. Brain-inspired computational intelligence via predictive coding. arXiv preprint wrxxi:2308.078777 2023.
[30] Benjamin Scellier. A deep learning theory for neural networks grounded in physics. arXiv preprint arXiv:2 103.0925,3,09955, 2021.
[31] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- -ased models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
[32] Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physical review letters, 61(3):259, 1988.
[33] David Stork. backpropagation biologically plausible? In International 1989 Joint Conference on Neural Networks, pages 241-246. IEEE, 1989.
[34] Liyua Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[35] Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and João Sacramento. Online learning of long-range dependencies. Advances in Neural Information Processing Systems, 36:10477-10493, 2023.
12

---

A Optimal Control
The classical way which Hamilton's 's equations are derived through Hamilton-Jacote Jaaiiionn-acob-BBellmaa theorem. So let enunciate this theorem in general setting. Here we use the notation (x, w) to stand for the whole state vector and p = (Pe- (Pr.Pw)- We will also denote with a the control parameters. Moreover to avoid cumbersome notation in this appendix we will override the notation on the symbols n and N and we will use them here to denote the dimension of the state and of the control parameters respectively.
A.1 Hamilton Jacobi Bellman Theorem
Consider the classical state model
v(t) 10==f(uu ((t)) E (to.
(25)
f: R" x RN x [to, T] + R" is a Lipschitz function, + act) is the trajectory of the parameters of the model, which assumed be a measurable function with assigned initial state y R", that is
y(4o) y"
(26)
Let us now pose A state y", we define the state trajectory, that indicate with + x(t 3, y to). the solution of (25) with initial condition (26).
: {a: [to, [] RN a is measurable} and given 3 E A, and given an initial
Now let us define cost functional C that want to minimize:
Co == ''' A(a(t), a, 3 (t)),
(27)
where A(a,", ,, 8) is bounded and Lipshitz Va RRN and Vs [to, TT. Then the problem
min Ca (a) aeA
(28)
a constrained minimization problem which is usually denoted control problem (. assuming that a solution exists. The first step to address our constrained minimization problem is to define the value function or cost to go, that is map v: R" x [tu, T] -> R defined as
v(E, s) = inf Cg.o(a), v(5,)) R" x [to. T aEA
and the Hamiltonian function H: IR x RD x [to T + R as
H P. s) == min {p f(E, a. + A(a. $)}, a
(29)
being- the dot product. Then Jamiiton-Jaaobi-Belle Theorem (Hamilton-Jacee Jacobi--ellnnnn
theorem states that
Let us assume that D denotes the gradient operator with respect Furthermore, let us assume that E CR" x [to, T), R) and that the minimum of Cg.s. Eq. (28). exists for every E RD and for every E [fo, T). Then v solves the PDE
vg(E. H(S, s), O,
(30)
(E, 8) E RD x (4o, T), with terminal condition u(5, 0, Ag R". Equation 30 usually referred to as Maniilion-Jacobi-Baee
equation.
Proof. Let s [4o, T') and & E R". Furthermore, instead of the optimal control let us constant control an (1) = a E IR-V for times E [8, + e] and then the optimal control for the remaining temporal interval. More precisely let us pose
02 E arg min C (( aEA
Now consider the following control
(t) (z(()
ift [s
(1(())
(31)
if +s,
13

---

Then the cost associated to this control is
stc
Ce.((a3)
+ (02
(32)
s4s
f s+E
ala,
dt
+ v(y(s a; (0,,,,)),00 8), :
By definition of value function we also have that v(s, s) & Cg.s(as). When rearranging this inequality, dividing by and making of the above relation have
v(y(s + t, 6, + + 8)
(33)
J. s+e
Ma,yllaa (l
Now taking the limit as + 0 and making use of the fact that y' (s, as, ((5,a,)) we get
V, 8) + Du(E, 8) [(S, a, 5) + A(a, E, 8) 2
(34)
Since this inequality holds for any chosen E IR. we say that
inf (u)(5.)) + Du(E, s) f(ç,a, + A(a, 5. s)} 2
(35)
aER>
Now we show that the inf is actually a min and, moreover, that minimum is O. To do this we simply choose a" E arg min Cg.o(a) and denote == (s), then
e(S, 9) = rst
(t) ()))
(36)
+ +
Then again dividing by and using that y (s;ss" {s,) = f(E, a s) we finally get
s=(5,s) + Dv(E, s) f(5,a', s) + A(a", .) =0 But since a" E R.V and knew that infagrN (u,(E,)) + Du(g, s) f(5,, a) + A(a,E,s)) 0 means that
(37)
inf (v,(E, + Do(g,8): f(5, 8) + Maa 8)
VERN'
(38)
min (va(a, Du(E,s): f(E, + A(a,E, =0.
aERN
Recalling the definition of we immediately see that the last inequality is exactly (HJB).
A.2 Hamilton Equations: The Method of Characteristics
Now let us define p(t) = Dv(y(t), t) so that by definition the value function p(T) = which gives (1). Also by differentiating this expression with respect to time we have
(t) = vs., (v(t), +
Σ Usus, t) yc. 2m1
(39)
Now since solves (30) if we differentiate the Hamilton Jacobi equation by & we obtain:
Urga (E. s) = - Hşu (5, Du(g, s), - Σ Hp, Dulc, s), DeLe, (E. s).
iml
Once we compute this expression on (y(t), t) and we substitute it back into 33 we get:
(t) - Hg. (v((), (o(x(1),())4) + Σ (e(() Ho: (v((), De((((),)) (,0))
UsLes (v(1), ().
1=1
Now if we choose yso that it satisfies ((t) Hp(y(t), t) the above equation reduces to ò = Hg(v(t),p(t) t).
Applying these equations to the Hamiltonian in we indeed end up with 12
14

---

B Proof of Theorem 2
From and the hypothesis on we have that
fe: = -;ōsk + Cko'
Σ
Wki
Σ
Wkm Smis
Eg = Lg, (20
jEpa(k)
mEpa(k)
(w. = cx
Σ
Wkmmmm
Σ
luws kVow
mEpa(k)
(Epa(k)
Then 12 becomes
c i: Ii + a Ejemate) WisZz)
úijj = -p's// (m/b))
j'z = c.p's Ek=d+t Emppaack CuP'oo' (Esepate) Wkitts) Wkmdms - cle (x), l)eb
1 (1) - Ek av (Emepa(s) Wkmmmm Enepack) ōuddgraa ck cliVarry (wn, l)
(40)
Now to conclude the proof it is sufficient to apply the following lemma to conveniently rewrite and switch the sums in the j equations.
Lemma 1. Let A be the set of the arches of a digraph as in Section 2 and let ②2 be true, then
A { (m, k) E A:KE {d I.....,.i. n (m,, k) A: m. nI....,
Equivalently we may say that Ek=d+1 Emepa(k) = Em= ExEch(m))
Proof. It is immediate consequences of the fact that the first neurons all parents of some neuron in {d + I,...,n} n} ((2 and that they do not have themselves any parents
C Proof of Proposition
The first equation of (66 is simple rewriting of the first expression in 14). utilizing the definition of activation from Eq. (15). We obtain the second equation (16) by combining the second and last equations in Eq
14
üig- -P (mcb) + p (mco")
The expression for p can be substituted from the last equation in Eq. (14), with p = mocob.y. Fi- nally, p'_
o (a,) from Eq. (15). To derive the second equation 16 we start by differentiating
Xr as defined in Eq. 15 obtaining:
X o"(as)
d'(as) pr.
p. o'aa)
d2 J2
We then substitute p poor (a;) )1 above and the third equation in 14 for pi. This equation, with all Pa terms converted to Au as per pt: = (as)( yields the exact expression for X in Eq. [6
D Proof of Proposition i
Let put): == o (Ejepat) (t) be. From the boundedness of o(-) we know that there exists
B > 0 such that [x(l)] B. Now we have
11( = (0)e att + 1 c-alt- 5 +B / e -al(t-r)
B
s <z,(0) + a
(1 e c < x2
a
15

---

W22
J/2
W/23
1021
12
U/42
1013
1//24
10
1334
1025
W332
11 11
1/35
107
0141
10.15
W55
374
U345
W154
J75
W55
Figure 5: Architecture used in the experiments. The red neuron is the one that used output and it is forced to follow the reference (target) signal.
E
Architectural and Algorithmic details
Figure illustrates the network architecture used in the experiments described in in Section 4.2 Algorithm| provides detailed explanation of the Hamiltonian Sign Flip method, which is also discussed in the same section and applied in all the experimental results presented.
Algorithm Hamiltonian Sign Flip. In red the change signs due to HSF. The locality of the method is evident from the loop time while the spatial locality depends the structure of each update rule for the states and costates. (what we propose is valid also for unevenly spaced data).
Init R rand, w = rand, 1e O, pa = Select and choose function while < T do Compute st using Eq. (24). at s'àt
it x f(z", w', t).
rss +- - (me/(mop")
it + s'
it. t Po (e w 1) - cl of t)
11. s'pt 1' + P'
if +- -v- fulxt w t) cl, (w, x. t)o..
+r x Tàt
w+ w + Tip
p² p + Tp'
P + TP
t = t
end while
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The main limitation of this paper are described in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification: All claims are properly stated and proofs are provided either in the main paper or in the appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The code with instructions to reproduce the experiments is provided.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: The code with instructions to reproduce the experiments is provided, no external data is needed reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssiicyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments only assess qualitative behaviour of a newly introduced learning rules.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Computing requirements of the experiment are so modest that any modern laptop can sustain it.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and in our opinion we are not in violation of any norm therein contained.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no direct clear foreseeable either positive or negative social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets, Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "1BAiRrx0GW", "reviewer": "Reviewer_FbKP", "summary": "This paper, titled \"Nature-Inspired Local Propagation,\" explores a novel learning framework that diverges from traditional machine learning methods, which heavily rely on large data collections and professional expertise. The authors propose a biologically inspired model emphasizing the critical role of temporal information processing. The framework assumes that learning and inference develop through a nature-based protocol of environmental interactions, where the agent uses limited memory buffers for processing information without recording the entire temporal stream. This approach, inspired by optimal control theory and principles of theoretical physics, emphasizes continuous learning through differential equations akin to natural laws.\n\nThe paper makes two main contributions. First, it introduces a local spatiotemporal pre-algorithmic framework inspired by Hamiltonian equations, showing that Backpropagation can be seen as a limit case of the proposed diffusion process at infinite velocity. This insight addresses the biological plausibility of Backpropagation by presenting a computational scheme that is local in both space and time, assuming the associated ordinary differential equations are solved as boundary problems. Second, the paper proposes a method for approximating the solution of Hamiltonian problems with boundary conditions using Cauchy's initial conditions, stabilizing the learning process through time reversal schemes related to the focus of attention mechanisms. The Hamiltonian Sign Flip (HSF) policy is experimentally validated for tracking problems in automatic control. While the proposed scheme optimally handles temporal settings and surpasses traditional algorithms like BPTT and RTRL, the authors note that real-world applications will require further development and substantial collaborative research.", "questions": "- The authors mention that \"Basically, the agent is not given the privilege of recording the temporal stream, but only to represent it properly by appropriate abstraction mechanisms. While the agent can obviously use its internal memory for storing those representations, we assume that it cannot access data collection. The agent can only use buffers of limited dimensions for storing the processed information. From a cognitive point of view, those small buffers allow the agent to process the acquired information backward in time by implementing a sort of focus of attention.\"  There are two claims on the conditions here: 1) no privilege if recording the temporal stream, and 2) no access to data collection, but only abstract representation from its internal memory. What is the buffer with limited dimensions that is used for storing processed information? What is the processed information and how is it encoded in a biological brain? I am wondering if this is a valid assumption neuroscientificly. \n\n- \"Interestingly, we show that the on-line computation described in the paper yields spatiotemporal locality, thus addressing also the longstanding debate on Backpropagation biological plausibility.\" what do you exactly mean by this sentence? The debate on the biological plausibility of BP has many aspects. It is not clear in this sentence, that the authors intend to address which aspect of the biological plausibility of BP? \n\n- line 73: \"In general we are interested in computational schemes which are both local in time and causal.\" It is not clear to me why causality matters. Is causality part of the solution constraints or problem constraints? And why is it necessary? Later on, we reverse this causality by considering reverse time. \n\n- c_i is the velocity constant to control the updates of neurons. Is the assumption c_i = c necessary since it is being used in most of the theorems? \n\n\n- Based on lines 74-77, and the definition of x(t) in lines 59-61, the underlying assumption is having inputs, parameters, and outputs as direct functions of (continuous) time rather than indexed by time. Is this a valid assumption? Do these values change by variable time, or do they change over time?\n\n\n- Is τ considered as a fixed variable for all vertices or its value can change for each vertex? Also, have you considered the setting where τ is also variable for each vertex?\n\n- line 52 says \"trajectory1 u: [0,+∞) → Rd.\" Lines 92 and 93 define \"vector u ∈ RN 7→ INi(u)\". This is not clear what IN is and if u is related to the previous definition of trajectories of input or not. Overall, in the spatial locality section, it is not clear to me what IN is and what is the intuition behind IN(u) and IN(w(t)). This section needs improvements.\n\n- Similarly, line 111 \"ϕ: [0, T] → R is a strictly positive smooth function that weights the integrated\" is confusing with the definition of \\phi earlier on in line 89? Are they conceptually related?\n\n- In equation 11, what is p. It is not also defined in appendix A. \n\n- line 155 \"where we have introduced the notation ai(t) to stand for the activation of neuron i\". Does a_i(t) represent pre-activations or activations?\n\n- how much is this approach scalable to the higher dimension of d and network architecture?\n\n- what are x, u, z in the plots? what is q? I did not understand the messages of the plots?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The computational model is clearly and generally defined based on the notations of computational graphs with natural limitations, which makes the theoretical achievements independent of the network architecture. This independence from architecture is an important feature in local learning approaches. \n\n- The paper is well-written and most of the time, the authors explained immediately what they meant after they used a technical term. Moreover, despite the paper being mostly theory-based, they tried to transfer some intuition behind their choices and assumptions. \n\n- I enjoyed the way the authors formalized the problem and the general idea of introducing the laws of learning in terms of Hamiltonian equations. However, there are some questions and assumptions that its validity is still not clear to me. I discussed them in the following sections. I would like to learn about those and possibly increase my score since I did enjoy the paper overall. \n\n- I liked the reasonable explanations of the assumptions when formalizing the problem and the computational schemes. For example, the assumption of locality in time and causality are both aligned with our understanding of time and the universe. They were even stepped further by assuming a more general case of l-locality in time, meaning that the values can depend on the values in the horizon l of the past (not necessarily Markovian!). Moreover, defining τ as a fixed variable shows that the authors thought about the possibility of an independent time scale of nodes, This implies that the authors were knowledgeable and careful about the practicality of the proposed method. \n\n- The authors were honest about the limitations of the work such as limitations regarding solving the ODE as a boundary problem.  \n\nOverall, I found the paper to be very engaging. I would like to improve my score once these questions and comments have been addressed.", "weaknesses": "Main weaknesses:\n- This paper introduces a pre-algorithmic approach, which provides some theoretical perspectives and hypotheses. The experimental evaluation and analysis of the approach are weak. The explanation of the graphs and experimental setup is not enough. Is the network 5 neurons with depth 1 and is linear or with sigmoid as activation? The explanation and notions used in the graphs are not clear. The goal of each plot and the conclusion is not clear. What were the authors trying to evaluate or show?\n\n- One of my biggest concerns is that it seems like there is an underlying assumption in the formalism of the paper where the authors assume the inputs, parameters and outputs are all a function of the variable of time. I am not so sure about the validity of this assumption. These values change based on other variables over time, meaning that they usually are indexed in time, rather than being a direct function of time. I would like to see more elaboration on this fundamental assumption and understand why it is valid. I agree that there could be defined a problem setting to see this dynamic, however, in most cases, this is not the case.\n\n- Another concern is around the assumption that the authors implicitly made in line 81, where they simplify equation 3. I am not sure why the causality in time is valid and necessary while later on\n\n\n- I believe the intuitive explanations of mathematical notions defined or borrowed from physics are not enough. I recommend modifying the write-up by defining a simple computational graph and showing the variables on it. Then discuss how possibly a Hamiltonian formulation can intuitively fit in for solving that graph using a spatiotemporal local learning rule. You can instead, enrich the references to cover the background better if there exist other works which are addressing the intuitive aspects of Hamiltonians in NNs. \n\n-  What metrics or benchmarks were used to measure the effectiveness of the HSF policy and the overall framework?\n\n- Appendix C is not elaborated enough to show how to go from eq.14 to eq.16. I suggest opening up the equations and differentiations and actually derive eq. 16 for the final version of the paper. Readers and reviewers would not want to spend time doing it on their own. \n\n- line 165 \"Proof. The proof comes immediately from (16).\" Justify better how you got eq. 17 directly from eq. 16. in the limit.\n\n- The references provided in this paper are not enough. The reader struggles to understand the background of different concepts used in the paper.\n\nSuggestions on referencing:\n- Line 35 \"The discrete counterpart, which is more similar to recurrent neural network algorithms that can be found in the literature, can promptly be derived by numerical solutions.\" The reference for the literature is missing here. \n\n- line 39: \"... thus addressing also the longstanding debate on Backpropation biological plausibility.\" a reference to the debates is missing. \n\n- The possibility of \"time reversal\"\n\n\nSome suggestions on writing and a few typos and grammatical errors: \n- line 10: \" ... when the speed of ... \"\n- line 30 \"..., those small buffers allow(s) the agent ...\" \n- line 51: \"Formally this amounts to determine to assign to each vertex i ∈ V a trajectory x_i that is computed parametrically in terms of the other neuron outputs and in terms of an environmental information, mathematically represented by a trajectory1 u: [0,+∞) → Rd.\" This sentence can be written more clearly. \n- line 81, \"Causality instead express(es) ...\"\n- line 88, \"the velocity constant (of) that controls the updates of\"\n- line 152 \"defined on chilren’s nodes\" --> children's\n- line 201, \"starting form\" --> from", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Explanation of the graphs and experimental setup**\nWe utilize a fully connected recurrent network with five neurons (see Figure 1 in the PDF rebuttal), each using $\\tanh$ activation functions. One neuron, designated as the output unit, aimed to approximate a given reference signal through a quadratic loss in the Lagrangian. Figures 1–3, demonstrate the effectiveness of the local spatiotemporal neural propagation algorithm and the sign flip policy. Figure 4 displays the behaviour of the Lagrangian and Hamiltonian over time.  In the figure caption “Track a highly-predictable signal” should be “Track a highly-unpredictable signal.”\n\n**Assumptions on temporal dependence**\nSee Q5 below\n\n**Assumption in line 8**\nWe assume for the derivation of the Hamilton equation that the state satisfies Equation (4). This equation is associated with multiple approximating sequences at discrete time, of the form in Equation (3). We emphasize the importance of causality because we aim to develop an algorithmic scheme that can be easily computed forward in time. However Section 3 does not rely on this implicit assumption, it only assumes~(4).\n\n**Defining a simple computational graph**\nYes, this facilitates understanding and will be included in the final version of the paper. In the attached PDF file we have illustrated the structure of the network used in the experiments, along with the main variables.\n\n**Metrics for HSF policy effectiveness**\nOur goal was to demonstrate the stabilizing effect and tracking capabilities of the HSF policy. We monitored the effectiveness of our approach using a quadratic loss function, which is commonly used in regression.\n\n**Appendix C**\nThank you again for the suggestion. We will expand the derivations in the proof in the final version of the paper.\n\n**Line 165**\nFrom the last line of (16) with $c_i=c$ we can divide both sides by $c$. In the limit $c\\to\\infty$ the only surviving terms are the ones that do not have $1/c$ in front; these are exactly the terms in Eq (17). We will expand the proof in the final version of the paper.\n\n**Suggestions on referencing (in order)**\n - Bertsekas, Dimitri. Abstract dynamic programming. Athena Scientific, 2022, pag. 2.\n - Francis Crick. The recent excitement about neural networks. Nature, 337:129–132, 1989. \n- Stork. \"Is backpropagation biologically plausible?.\" International 1989 Joint Conference on Neural Networks. IEEE, 1989.\n- Timothy P. Lillicrap Geoffrey Hinton, Backpropagation and the brain, Nature Reviews - Neuroscience, 2020\n - Evans, Lawrence C. Partial differential equations. Vol. 19. American Mathematical Society, 2022. Pag.597\n\n**Answers to questions** \n\n**Q1.** By “processed information,” we refer to the data stream $u$. The internal memory representation in our model is given by the state variables (outputs and weights). Some justification for our assumptions may be found in the way representational systems are realized from a neurobiological perspective.\n\n - Byrne, John H. Learning and memory: a comprehensive reference. Academic Press, 2017. Pag. 228–230.\n\n**Q2.**  We address two issues: the “update locking problem” and the often-overlooked issue of infinite signal propagation speed in neural networks. A clarifying sentence will be added.\n\n\n**Q3.**  We aim to develop a causal learning algorithm that processes information forward in time. Confusion may arise from continuous-discrete transitions in Section 2. We will make efforts to address it further.\n\n**Q4.** The assumption is not necessary; it was used for simplification. We only require the condition  $c_i/c \\to 1$  as  $c \\to \\infty$  for all  $i$ if we want a reduction to backpropagation.\n\n**Q5.** We assume  $ t \\mapsto u(t)$ given function of time. Outputs, while also functions of time, are subject to dynamic constraints Eq.(5), and indeed depend on variables (neurons, inputs, and weights). Our proposal emphasizes the explicit temporal dependence of the parameters, modeling the continual adaptation necessary for continuous learning. We advocate shifting from finding a set of optimal parameters to searching for optimal weight trajectories. This approach reflects the ordinary cognitive perspective where learning and inferential processes are not separate.\n\n**Q6.** All the results holds if the spacing between points in the temporal partition is non-uniform, so $\\tau\\to\\tau_n$.\n\n**Q7.** Yes, we agree that we need to be more clear. At any rate the $u$ in line 52 is the input signal while the bold **u** in line 92 and 93, is a generic vector that lives in the parameter space. ${\\rm IN}^i(w)$ selects the components on the weights $w$ associated to arcs that point to neuron $i$. We will change the notation.\n\n**Q8.** No, they are not related. The function in line 89 is a capital $\\Phi$ and it specifies the structure of the neuron. The one on line 111 is an overall factor to the lagrangian that weights the contributions of different time steps to the overall cost. We will use more different symbols.\n\n**Q9.** It is a generic vector that has the same dimension of $x$. You are right, we need to add a definition.\n\n**Q10.** Since in the literature the term activation is not always used consistently the best way to answer your question is to say that $a_i$ are the quantities defined in Eq (15), that may be what you refer to as pre-activations.\n\n**Q11.** From a computational point of view the approach is equivalent to GD + backpropagation and so it can in principle be used on networks of the same scale that are now used in deep learning.\n\n**Q12.** Thanks, for the question. In the plots $x$ is the output of the output neuron (in the new attached figure it is $x_1$), $u$ is the input signal, and $z$ is the reference signal. The parameter $q>0$ weights the term of the lagrangian that enforces the fit between $x_1$ and $u$. The purpose of the plots is to show that the usage of Hamilton’s equation with the described sign flip prescription is effective in solving an online tracking problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Qzuw7ATIu", "reviewer": "Reviewer_FXeY", "summary": "The paper proposes a spatially and temporally local alternative to backprop for training RNN. The paper first proposed viewing learning as minimizing a functional of a variational problem using Hamiltonian equations. The standard backprop can be seen as a special case when the velocity in the functional is set to infinity. Then, the paper proposes an alternative to solving the functional based on Hamiltonian Sign Flip that does not require going backward in time to compute the weight update as in conventional backprop. This allows online processing of inputs without needing a separate backprop phrase.", "questions": "Am I correct that the proposed algorithm is based on (23) with zero initial conditions and s(t) defined in (24)? In case the loss function is the difference between the label and output neuron's value in the final time step, and the input is a constant image, will this algorithm work as desired? How could the weight be updated when the label is not accessible before the final time step?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper gives a new perspective on how backprop can be seen as a special case of solving a variational problem with infinite velocity\n- The paper proposes an alternative to solving the variational problem without requiring going backward in time as in backprop", "weaknesses": "- Lack of clarity: The paper can be written less technically and more intuitively so it is accessible to a broad audience. I found it difficult to understand and struggled to fully understand how the sign flips remove the need to go backward in time. It would also be much better if the pseudo-code of the final algorithm were presented directly, which would allow for more reproducible results. \n- Lack of experiment: even though the paper focuses on a theoretical understanding of the learning rule, it is essential to have more standard experiments to understand the difference with the conventional backdrop. For example, evaluation on MNIST and comparing it with backprop, using more neurons instead of just 5, is necessary for readers to understand its border applicability.\n- Related work: There is a lot of related work on biologically plausible learning rules that are temporally local. A related work section and discussion on the relationship with these previous works will allow readers to understand its contribution much more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty of our approach. We will address the comments and questions raised.\n\n*It would also be much better if the pseudo-code of the final algorithm were presented directly, [...]*\n\nWe agree that the presentation of the final algorithm that we are using could be improved. We will add an appendix with the pseudo-code that you can read in the attached PDF (at the end of the overall rebuttal).\n\n*Lack of experiment [...] is necessary for readers to understand its border applicability.*\n\nThis paper proposes a natural learning scheme that promotes environmental interactions. It covers theoretical foundations, and the experimental results are expected to validate the consistency of the theory rather than achieving high performance on classical benchmarks. We acknowledge the importance of conducting similar experiments. The proposed test can be conducted for instance by generating an appropriate data stream from MNIST, where each pattern is presented at the input for a fixed amount of time. However, this requires a considerable amount of experimental work and is beyond the scope of this paper. It is worth mentioning that the MNIST experimental setting described above is not the primary focus of the theory and that experimental work in this area is still in its early stages. We also would like to point out that one of the topic from the call for paper is \"theory (e.g., control theory, learning theory, algorithmic game theory)\", which we believe is coherent with the spirit of this paper.\n\n*Related work [...]*\n\nWe will add a “Related Work” section either after the introduction or as a subsection of it. In the overall rebuttal comments, we have included a draft of this section.\n\n*Am I correct that the proposed algorithm is based on (23) with zero initial conditions and s(t) defined in (24)? [...] How could the weight be updated when the label is not accessible before the final time step?*\n\nYes, you are correct but with random initialisation on the state. If supervision is not provided until the end of the temporal interval, changes in the weight during the middle of the interval will not be informed by the label but only by regularisation terms; all the Hamiltonian equations are the same.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5V3vsB7oJP", "reviewer": "Reviewer_GgV1", "summary": "The authors introduce a new model for describing the evolution of weights in a neural network. They do so by defining the problem as a directed graph and reformulating it to derive a Hamiltonian which they can then apply Hamilton's principle to to solve. This solution yields a set of differential equations from which they argue they can read off the dynamics of weights. They further show this form can be reduced to standard gradient descent by letting the velocity of the system go to infinity.", "questions": "1. Which elements of your derivation depend on control theory? From what I followed, you introduce a Lagrange multiplier problem, derive a Hamiltonian, and then look for the minimum of the action. I understand that this process is followed in control theory, however, is there something I missed that relates it directly to some element of control? \n2. What is the overall conclusion regarding data requirements? I lost track of this message towards the end of the paper. Normally, evolution equations will depend on data in some way, whether you look for evolution in a parameter space or some interacting picture, in the introduction the authors argue that large data-sets are not needed. I missed how the derivation ties back to this, perhaps you can make that more clear to me here.\n\nI also had some spelling/grammar points for future versions of the paper:\n\nl.16: strongly rely on\nl.39: Some citations here would be helpful\nl.50: amounts to determining how\nl.51: in terms of environmental information (remove the an)\nPage 2 footnote line 2: domain and co-domain\nPage 5 footnote line 1: until now\nl.201: Starting from\nl.258: define a Hamiltonian track\nl.298: leads to the interpretation of back...\n\nThis list is not exhaustive, just what I marked.", "limitations": "Application to more relevant models would be very helpful. Despite the statement that this should be treated only as a theoretical contribution, it was not clear to me how I could use the conclusions. Therefore, either some demonstration or a clearer conclusion in that regard would be very useful.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper takes a rigorous approach to understanding network dynamics and it does so with a relatively small set of assumptions. The mathematics, while slightly over-dominating, appear to be well derived and lead to consistent conclusions.", "weaknesses": "The authors find a set of differential equations which minimizes their derived action. What I struggled to see was how this yielded new approaches to understanding real dynamics. I appreciate that the author/s states in the conclusion that more work needs to be done to apply this to more applicable machine-learning problems, however, as a machine-learning practitioner I am not sure where I would start. I think this is something that could be cleared up in the paper. Exactly what the desired outcome is of the work, rather than just, a mathematical form, and how it can be applied. This all essentially boils down to, I think there would be a simpler way to present the results of the paper without the, at times, excessive complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for having appreciated our work. In what follows we will address the comments and questions raised.\n\n*I appreciate that the author/s states in the conclusion that more work needs to be done to apply this to more applicable machine-learning problems, however, as a machine-learning practitioner I am not sure where I would start. I think this is something that could be cleared up in the paper. [...]*\n\nThanks for pointing this out—it’s a really important observation! The learning framework we have in mind while developing these ideas, and one of the main motivations for this work, is lifelong learning from a continuous, possibly infinite, stream of data. We decided to give this work a more technical style because, as we expressed in the conclusions, we feel it is necessary to establish some precise results to build upon.\nOne prototype of the learning problems we believe this approach is best suited for is learning from a very long video stream. In this case, the input stream $u$ will be the video stream, and a simple Euler method for Hamiltonian equations will readily provide an “optimizer” for the weights of a recurrent neural network (RNN). We will do our best to incorporate explicit statements on the desired outcomes of the work in the revised version of the paper.\n\n*Which elements of your derivation depend on control theory? [...]*\n\nYou are perfectly right. One of the main results of the paper, Theorem 1, is derived as an application of standard control theory. Indeed, what we are trying to do is find optimal trajectories for the weights of a continuous-time RNN by controlling the velocity of the weights. This is expressed in the first paragraph of Section 3.1. In particular, the control variables are denoted by **v** in Equation (10). The adoption of this theory makes it possible to define a new spatiotemporal neural propagation scheme that generalise backprogatation.\n\n*What is the overall conclusion regarding data requirements? [...] I missed how the derivation ties back to this, perhaps you can make that more clear to me here.*\n\nYes, precisely as you are saying, data enters the evolution equations through the input signal $t\\mapsto u(t)$. For instance, when processing a video stream, $u$ is the video itself, and $u(t)$ is the frame of the video at time $t$. Notice that this signal is used in the computation of the outputs of the neurons (see Eq. (5)), as one would expect: $u$ updates the input neurons, which in turn update their children, and so on. For this reason, as noted in the introduction, this theory is applied in the context of lifelong/continuous learning, where the main assumption is that data is handled as a stream of information instead of a static training set. This means that at each time $t$ we only need to access  $u(t)$ and we neither need to store nor to access data collections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are pleased to see the detailed analysis and the insightful list of comments and criticisms, which are greatly appreciated, along with the time dedicated to reviewing our work. We are confident that these will significantly improve the quality of our paper.

Since more than one reviewer commented on the necessity of additional references, we followed Reviewer FXeY’s suggestion to reorganize the introduction and include a standalone “Related Work” section. Here is a draft of the content for this new section:

**Optimal control.** The main focus of optimal control is the study of minimality problems for dynamical systems [1, 2]. The two main complementary approaches to this problem are the Pontryagin Maximum Principle [3] and dynamic programming. As a general minimization problem, both approaches intersect significantly with the calculus of variations [4]. Optimal control for discrete problems is also a classic topic [5, p. 2].

**Neural ODE.** Recently, in [6] and many subsequent papers [7, 8], results from optimal control have been used to define learning algorithms based on differential relations. However, these approaches differ significantly from the continual online learning considered in the present work, as the time in the class of ODEs they study is not related to the input signal that describes the flow of the learning environment.

**Online.** Several works have proposed formulating learning problems online from a single stream of data [9, 10]. The classical approach for learning RNNs online is Real-Time Recurrent Learning (RTRL) [11]. Since then, many approaches have been proposed to reduce the high space and time complexities associated with the progressive update of a Jacobian matrix [12]. In our method, no Jacobian matrices are stored; therefore, the proposed method is neither a generalization nor a reformulation of RTRL or related approaches like [13].

**Nature-inspired computations.**  The major difference between our approach and other attempts to address the biological plausibility of backpropagation is that we propose a theory fully based on temporal analyses in the environment and the concept of learning over time. Many other classical [14] and recent approaches [15, 16, 17, 18, 19] are inspired by brain physiology, even if they share some locality properties described in this paper. Similarly, the vast majority of works that discuss the biological plausibility of backpropagation [20, 21, 22] overlook the role of time as we present it in this work.
In contrast, we propose laws of neural propagation where the neural connections are updated over time, resembling natural processes.

[1] Bardi, Martino, and Italo Capuzzo Dolcetta. Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations. Vol. 12. Boston: Birkhäuser, 1997.

[2] Cannarsa, Piermarco, and Carlo Sinestrari. Semiconcave functions, Hamilton-Jacobi equations, and optimal control. Vol. 58. Springer Science & Business Media, 2004.

[3] Gamkrelidze, R. V., Lev Semenovich Pontrjagin, and Vladimir Grigor'evic Boltjanskij. The mathematical theory of optimal processes. Macmillan Company, 1964.

[4] Giaquinta, Mariano, and Stefan Hildebrandt. Calculus of variations II. Vol. 311. Springer Science & Business Media, 2013.

[5] Bertsekas, Dimitri. Abstract dynamic programming. Athena Scientific, 2022.

[6] Chen, Ricky TQ, et al. "Neural ordinary differential equations." Advances in neural information processing systems 31 (2018).

[7] Kidger, Patrick, et al. "Neural controlled differential equations for irregular time series." Advances in Neural Information Processing Systems 33 (2020): 6696-6707.

[8] Massaroli, Stefano, et al. "Dissecting neural odes." Advances in Neural Information Processing Systems 33 (2020): 3952-3963.

[9] Mai, Zheda, et al. "Online continual learning in image classification: An empirical survey." Neurocomputing 469 (2022): 28-51.

[10]  Wang, Liyuan, et al. "A comprehensive survey of continual learning: theory, method and application." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[11] Irie, Kazuki, Anand Gopalakrishnan, and Jürgen Schmidhuber. "Exploring the promise and limits of real-time recurrent learning." arXiv preprint arXiv:2305.19044 (2023).

[12] Marschall, Owen, Kyunghyun Cho, and Cristina Savin. "A unified framework of online learning algorithms for training recurrent neural networks." Journal of machine learning research 21.135 (2020): 1-34.

[13] Zucchet, Nicolas, et al. "Online learning of long-range dependencies." Advances in Neural Information Processing Systems 36 (2023): 10477-10493.

[14] Rao, Rajesh PN, and Dana H. Ballard. "Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects." Nature neuroscience 2.1 (1999): 79-87.

[15] Salvatori, Tommaso, et al. "Brain-inspired computational intelligence via predictive coding." arXiv preprint arXiv:2308.07870 (2023).

[16] Millidge, Beren, Alexander Tschantz, and Christopher L. Buckley. "Predictive coding approximates backprop along arbitrary computation graphs." Neural Computation 34.6 (2022): 1329-1368.

[17] Ororbia, Alexander, and Ankur Mali. "The predictive forward-forward algorithm." arXiv preprint arXiv:2301.01452 (2023).

[18] Ororbia, Alexander G., and Ankur Mali. "Biologically motivated algorithms for propagating local target representations." Proceedings of the aaai conference on artificial intelligence. Vol. 33. No. 01. 2019.

[19] Hinton, Geoffrey. "The forward-forward algorithm: Some preliminary investigations." arXiv preprint arXiv:2212.13345(2022).

[20] Francis Crick. The recent excitement about neural networks. Nature, 337:129–132, 1989. 

[21] Stork. "Is backpropagation biologically plausible?." International 1989 Joint Conference on Neural Networks. IEEE, 1989.

[22] Timothy P. Lillicrap Geoffrey Hinton, Backpropagation and the brain, Nature Reviews - Neuroscience, 2020

Meta-review: This paper proposes a new, pre-algorithmic viewpoint on learning as an optimal control problem in real time. That is, the parameter trajectory over time is governed by a Hamiltonian that solves the Hamilton-Jacobi-Bellman equation. By requiring that the resulting equations are both causal and local (both in time and in the sense of the computational graph), the authors then derive a set of equations governing learning. Unfortunately, these equations involve boundary conditions at the _end_ of the time interval, violating causality. In order to restore causality, the authors propose a a new set of dynamics based on sign flips and conjecture that this is an optimal solution.

All reviewers agreed that the idea is interesting and that the pre-algorithmic, theoretical approach is valuable. However, reviewers also agreed that the paper is quite difficult to read, both because it is particularly heavy on formality and because little effort is expended to give general machine learning readers an intuition for the problem. Likewise, the experiments were judged as weak, even for a theoretical paper, and this objection is not negligible, since the actual learning equations proposed are not proven but only conjectured to be optimal.

However, on balance, this seems like an interesting work likely to spark discussion. Accept.
Decision: Accept (poster)