Neural optimal feedback control with local learning rules
Authors: Johannes Friedrich
Siavash Golkar
Shiva Farashahi
Alexander Genkin
Anirvan M. Sengupta
Dmitri Chklovskii
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: optimal feedback control
Kalman filter
motor control
neural circuit

Abstract: A major problem in motor control is understanding how the brain plans and executes proper movements in the face of delayed and noisy stimuli. A prominent framework for addressing such control problems is Optimal Feedback Control (OFC). OFC generates control actions that optimize behaviorally relevant criteria by integrating noisy sensory stimuli and the predictions of an internal model using the Kalman filter or its extensions. However, a satisfactory neural model of Kalman filtering and control is lacking because existing proposals have the following  limitations: not considering the delay of sensory feedback, training in alternating phases, requiring knowledge of the noise covariance matrices, as well as that of systems dynamics. Moreover, the majority of these studies considered Kalman filtering in isolation, and not jointly with control. To address these shortcomings, we introduce a novel online algorithm which combines adaptive Kalman filtering with a model free control approach  (i.e., policy gradient algorithm). We implement this algorithm in a biologically plausible neural network with local synaptic plasticity rules. This network, with local synaptic plasticity rules, performs system identification, Kalman filtering and control with delayed noisy sensory feedback. This network performs system identification and Kalman filtering, without the need for multiple phases with distinct update rules or the knowledge of the noise covariances. It can perform state estimation  with delayed sensory feedback, with the help of an internal model. It learns the control policy without requiring any knowledge of the dynamics, thus avoiding the need for weight transport. In this way, our implementation of OFC solves the credit assignment problem needed to produce the appropriate sensory-motor control in the presence of stimulus delay.

Neural optimal feedback control with local learning rules
Johannes Friedrich
Siavash Golkar'
Shiva Farashahi
Alexander Genkin
Anirvan M. Sengupta 2,3,4
Dmitri B. Chklovskii 1,5
Center for Computational Neuroscience, Flatiron Institute
Center for Computational Mathematics, Flatiron Institute
3 Cenntrr for Computational Quantum Physics, Flatiron Institute
DDepartment of Physics and Astronomy, Rutgers University
Neuroscience Institute, NYU Medical Center
(jfriedrich, sgolkar, sfarashahi, ott.aaaccciiiiiioooaaaooooooaatt org
(alexander genkin, anirvans physics))ggaill com
Abstract
A major problem in motor control is understanding how the brain plans and executes proper movements in the face of delayed and noisy stimuli. A prominent framework for addressing such control problems Optimal Feedback Control (OFC). OFC generates control actions that optimize behaviorally relevant criteria by integrating noisy sensory stimuli and the predictions of an internal model using the Kalman filter its extensions. However, satisfactory neural model of Kalman filtering and control is lacking because existing proposals have the following limitations: not considering the delay of sensory feedback, training in alternating phases, and requiring knowledge of the noise covariance matrices, as well as that of systems dynamics. Moreover, the majority of these studies considered Kalman filtering in isolation, and not jointly with control. To address these shortcomings, we introduce novel online algorithm which combines adaptive Kalman filtering with a model free control approach (i.c., policy gradient algorithm). We implement this algorithm biologically plausible neural network with local synaptic plasticity rules. This network performs system identification and Kalman filtering, without the need for multiple phases with distinct update rules the knowledge of the noise covariances. It can perform state estimation with delayed sensory feedback, with the help of an internal model. learns the control policy without requiring any knowledge of the dynamics, thus avoiding the need for weight transport. In this way, our implementation of OFC solves the credit assignment problem needed to produce the appropriate sensory-motor control in the presence of stimulus delay.
Introduction
The sensorimotor control system has exceptional abilities to perform fast and accurate movements in variety of situations. To achieve such skillful control, this system faces two key challenges: (i) sensory stimuli are noisy, making estimation of current state of the system difficult, and (ii) sensory stimuli is often delayed, which if unaccounted, results in movements that are inaccurate and unstable [[]. Optimal Feedback Control (OFC) has been proposed as solution to this control problem 2 B) OFC approaches these problems by building an internal model of the system dynamics, and using
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

this internal model to generate control actions. OFC often employs Kalman filtering to optimally integrate the predictions of this internal model and the noisy/delayed sensory stimuli.
Because of the power and flexibility of the OFC framework, biologically plausible neural architectures capable of building such internal models has been under active investigation. Specificallyy, earlier works used attractor dynamics implemented through a recurrent basis function network [4] or line attractor network (5) implement Kalman filters. Kalman filtering and control has also been implemented through different phases of estimation, system identification and control 6 and more recently, using a particle filtering method for Kalman filtering 7
Nonetheless, these works suffer from major limitations. Importantly, none of these considered that sensory feedback is delayed 44 6
7 8) although it has been prominent in the original compurational-level OFC proposal 22 or merely considered the case of Kalman filtering, and not 4 S
5
g These works also required knowledge of the noise or obtained in a separate 'offine sensor' mode [6] Moreover, many of these works lack biological plausibility and realism one would expect from a viable model of brain function 4 8). Crucially, biological plausibility requires the network to operate online, (i.e. receive stream of noisy measurement data and process them on the fly), and also requires synaptic plasticity rules to be local (i.e. learn using rules that only depend on variables represented in pre and postsynaptic neurons and/or on global neuromodulatory signals). Lastly, several of these models suffer from combinatorial explosion as the dimensionality of the input grows 55 require or require separate learning and
the combination of it with control
covariances, either a priori [ 5
running an inner loop until convergence each time step g 6
execution phases (6) cf. Table [
We address these shorteomings and present complete neural implementation of optimal feedback control, thus tackling an open issue in biological control [9. In this model, which we call Bio-OFC, the state space. the prediction error []0 (i.e., the mismatch between the network's internal prediction and delayed sensory feedback), and the control are represented by different neurons, cf. Fig. D The network also receives scalar feedback related to the objective function, as a global signal, and utilizes this signal to update the synaptic connection according to policy gradient method [2 13) To test the performance of our network, we simulate Bio-OFC in episodic (finite horizon) tasks (e.g.. a discrete-time double integrator model, hand reaching task [ and simplified fly simulation).
Summary of contributions:
We introduce Bio-OFC, a biologically plausible neural network that combines adaptive model based state discovery via adaptive Kalman filtering with model free control agent.
Our implementation does not require knowledge of noise covariances nor the system dy- namic, considers delayed sensory feedhack, and has no separate learningeexecution phases.
Our model-free control agent enables closed-loop control, thus avoiding the weight transport problem, a challenging problem even in non-biological control. [44
Table 1: Limitations of previously proposed neural implementations of OFC. Presence absence of different properties in previously proposed neural models, and their comparison to Bio-OFC. Guide symbols: v: true, X: false, vX: partially true, N/A: not applicable.
6
g
Bio-OFC
delayed sensory feedback control included
X
X
X
X
/
X
X
noise covariance agnostic
/
X
X
X
online system identification
X
/
X
VX
local learning rules tractable latent size absence of inner loop
N/A
N/A
X
X
V
X
/
X
X
single phase leaming/cxecction
X
N/A

---

System ID And State Prediction
Motor Control
Learning Rules
Kalman filtering with delay
Palicy gradient with eligibility traces
AA, x Le, #
AB, x Leg u
Efferent copy
Ah x Le of
Prediction
B
AC; R
Sensory
Le
Motor
K
2, = 2, + E
A
Action
G = mG + 2
C
Costic)
AK; -G,
Figure 1: The circuit and learning rules of the Bio-OFC algorithm. Our circuit is comprised of two main parts. First (in blue), the circuit performs Kalman filtering. Then (in red), the circuit performs control using policy gradients with eligibility traces. Triangular arrowheads denote synaptic connections and the flat arrowhead denotes the modulatory effect of the cost signal.
2
Background
We review classical Kalman estimation and control in this section, using boldface lowercase/uppercase letters for vectors/matrices and I for the identity matrix.
2.1 Problem formulation
We model the environment as a linear dynamical system driven by control input and perturbed by Gaussian noise. The true state of the system hidden and all the animal has access to are the observations that are assumed to be linear functions of the state corrupted by Gaussian noise.
dynamics:
L(+1 = Axx + Bu, +
y/ = Cæ. + wi Here V. (0; V) and w, (O, W) are independent Gaussian random variables and the initial state has a Gaussian prior distribution Ey ~ N2 En).
observation:
(2)
The goal is to estimate the latent state è in order to design control that minimizes expected cost
T
expected cost:
E
Σ
cla,, u.)
(3)
control:
Ur k(+)) arg arg min
where c(x,, u.) is the instantaneous cost associated with state æ, and action u4 As the environment dynamics is not known to the animal a priori, the parameters A, B, C must be learned online.
2.2 Kalman estimation and control
The Kalman filter 16 is an estimator of the latent state ã;; (and its variance) via weighted summation of the current observation and the prediction of the internal model based on prior measurements. However, in biologically realistic situations the sensory feedback is always delayed. This means that the control signal u, has to be issued, and thus the state æ, estimated, before ye has been observed. The appropriately modified Kalman filter computes the posterior probability distribution of æ; given observations y:-r, ...; Yo where T 2 is the delay. We start here with case of 7 1 for which the recursive updates of the mean Ø; and the variance E,
dr+1 Ad; Bu, L, Ly C..)
(5)
Σ (A L, +
(6)
where L, is known the Kalman gain matrix which optimally combines the noisy observations y/ with the internal model and is given by
L, A (CE,CT + +
3

---

The Kalman filter is optimal in the sense that it minimizes the mean-squared error E with prediction error (innovation) e, = y/: Ct.,
The output feedhack law, also known as policy, (4) simplifies if the cost J is quadratie in ua and ay -
(8)
and is known as linear-quadratic regulator (LQR). The control gain, K, is found by solving matrix Riccati equation (cf. Supplementary Material). Linear policies have been successfully applied to a variety of control tasks 17
3 Neural network representation for optimal feedback control
3.1 Inference
For now, let us assume that the system dynamics. the Kalman gain and the control gain A, B,C, L. and K are constant and known. Then, the latent state o can be obtained by the Kalman estimator Eq. (5) This algorithm naturally maps onto the network in Fig. | and Supplementary Fig. (STJ with neural populations representing &, e =- y Că. and u that are connected by synapses whose weights represent the elements of the matrices A, B, - -C and L. The computation of the control variable u according to Eq. (8) can be implemented by synapses whose weights represent the elements of the matrix K.
If the sensory stimulus delay is in the previous section. In case r > 1, the latent state must be recomputed throughout the delay period which requires a biologically implausible circuit (cf. Supplementary Fig. (ig. To overcome this, we adapt an alternative solution from online control [18 [9]. Specifically, we combine delayed measurements with the similarly delayed latent state estimation. Such inference can be performed by the network of the same architecture and adjusting the synaptic delay associated with matrix C' to match with the sensory delay, in accordance with the following expression:
our network implements the Kalman prediction reviewed
L Ad,- + Bu; + L (y++1 -(
As we show in Results (Fig. 3 this reduces predictive performance only modestly compared the biologically unrealistic scheme. More details on the inference process and the temporal order in which our recurrent network performs the above steps is shown in Supplementary Fig. STB.
3.2 Learning
3.2.1
System identification and Kalman gain
Next, we turn our attention to the system identification/learing problem, which was not addressed by some of the previous proposals, cf. Table Given sequence of observations {3o. .rr). we use a least squares approach (20) to find the parameters that minimize the mean-square prediction error + EiT e,_ et. We perform this optimization in an online manner, that is at each time-step ++ 1, after making the delayed observation Yt+1-7: update the parameter estimates Á, B, Ć using steps that minimize e, e;, assuming that the state estimate and actions corresponding to prior observations (c.g. Φ;-7, Ut- etc.) are fixed. To obtain L we would like to avoid solving the Riccati equation 66) as it requires matrix operations difficult to implement in biology. So, we use the same optimization procedure to update the Kalman gain L. Using Eq. (9) and explicitly writing out the matrix/vector indices as superscripts yields the following stochastic gradient with respect to A:
θ 1 Σ
det DAV
2 (ut ( 1- -E CKO a (1+1--)-
DA (et) =
Σ k
Σ
2
E
DAN
k
(E.n Alm sm
+ En Bluui uf + Ep L'be.)
Σ
chi
0A.
Σ
Chi am Σ C.Chii
(10)
k.

---

Performing similar derivations for the other synaptic weights, our optimization procedure would rely on the following stochastic gradients:
- - = C e
VBže, C' euu Vc e, e,
(11)
C
(12)
This yields a classical Hebbian rule between (a memory trace of) presynaptic activity 2 and postsynaptic activity e, for weights C. However, it suggests non-local learning rules for A, B. L, which runs contrary to biological requirements. We can circumvent this problem by replacing C' with L, which corresponds to aft-muuliiliiaation of the gradients with positive definite matrix (see Supplementary Material Sec. 4 This still decreases the mean-square prediction error under some mild initialization constraints C' and and yields local plasticity rules, cf. Fig.
A x Le, at-T AL, Le; ex-
(13)
AB, x Le u.
(15)
(14)
AC, e, 2
(16)
where the input current Le, is locally available at neurons representing æ. The first three rules are local, but non- Heb-Heb capturing correlations between presynaptic activity Ø;-7, U Ct-r and postsynaptic current Le,. Note that these updates do not require knowledge of the noise covariances V and W, an advantage over previous work, cf. Table
3.2.2 Control
Next, we consider optimal control a neural implementation of which was missing in most previous proposals, cf. Table Traditionally, optimal control law is computed by iterating matrix Riccati equation, cf. Supplementary Material, posing a difficult challenge for a biological neural implementa- tion (but see (21)) To circumvent this problem,
propose to learn the controller weights K using policy gradient method [2, [2 instead. Policy gradient methods directly parametrize a stochastic controller K (u) Representing the total cost for a given trajectory T c(T), they optimize the pa- rameters K by performing gradient descent on the expected cost J = Ex [c(T)] E
2 Ct
VKJ = [
d[] = Exx V (o))
(17)
T VV Vx log ( 2.))
= Exx
(18)
The term in square brackets is an unbiased estimator of the gradient and can be used to perform stochastic gradient decent. As already hinted at by (12), due causality, costs o are affected by later controls u. > t, and the variance of the estimator can be reduced by excluding those terms, which yields
AK
Σ (=0
Σ
V log T (1gg(()))))
(19)
We simply keep an eligibility trace of the past, Z. E. V log ग ( x.), and perform parameter updates AK x -c each time step L. A similar update rule has been suggested by 233 24 for the infinite horizon case. Global convergence of policy gradient methods for linear quadratic regulator has been recently studied by Fazel et al. (25).
We assume the output of neurons encoding control perturbed by Gaussian noise
W
K E. with & ~~N,000)
(20)
The synapses are endowed with a synaptic tag (26 Z, an eligibility trace that tracks correlations between pre-synaptic activity and post-synaptic noise &. It reset zero at the beginning each trajectory, though instead of hard reset it could also softly decay with time constant of the same order (TT) as trajectory duration (27). The weight update assigns cost e to the synapses according to their eligibility Z.. The cost is c.g. provided by diffuse neuromodulatory signal such as dopamine.
5

---

The optional use of momentum m E [0, adds low-pass filter to the synaptic plasticity cascade:
Z: Z + E (= En V log To (xss
(21)
G. = mG 1 + + c.
(22)
A G₁
(23)
4 Experiments
In this section we look at three different experiments demonstrate various features of the Bio-OFC algorithm. In Sec. 4.1 we look at a discrete-time double integrator and discuss how our approach performs not only Kalman filtering (learning the optimal Kalman gain), also full system-ID the open-loop setting (system-ID followed by control) as well as in the more challenging closed- setting (simultaneous system- and control). In each case, we provide quantitative comparisons and discuss the effect of increased delay. In Secs. 4.2 and 4.3] we apply our methodology to two biologically relevant control tasks, that of reaching movements and flight.
4.1 Discrete-time double integrator
As a simple test case, we follow along the lines of [28] and consider the classic problem of discrete-time double integrator with the dynamical model
.01 0 01) E++1 ~ ((Ax; + Bu, V) where A . B - (i) V= = (0l
(24)
Such system models the position and velocity (respectively the first and second components of the state) of unit mass object under force u. As an instance of LQR, we can try to steer this system to reach point (0, 0) from initial condition wo = () without expending much force:
Q -0) R=1, T= 10
xz Q +
Σ of
where
(25)
We assume that the initial state estimate T. is at the true initial state (áo Cxx))
We go beyond LQR (cf. Supplementary Fig. (22A) and assume is not directly observable (cf. Supplementary Fig. B,C), but we merely have access to noisy observations, y ~ (Crr W). We consider observation models, one where the state is only observed with uncorrelated noise,
and one where the observation noise covariance is not diagonal and an additional mixture of the state components observed:
LDSI: C= (6 :). W .004
111 .5
25
LDS2: C =
-1
W
.09
25
0
(26)
.04
We denote these two systems as linear dynamical systems and 2 (LDSI and LDS2).
Learning the Kalman gain. A major advantage of our work is that it does not require knowledge of the covariance matrices V and W to determine the Kalman filter gain L. We studied this using LDS1 in scenario where the observation noise varies; changing the covariance matrix W from diag( .25 to diag(.04,.00) after 2500, to diag( (.0a, .01) after 5000, and back diag( (04, .25) after 7500 episodes. In this experiment we fix A, B, C' at the ground truth, and initialize L at the optimal value for W (.0, .25), and updated the latter according to Eq. 14p. Fig. shows how the elements of the filter matrix L adapt in time, to optimize performance measured by the mean squared prediction error. The learning rate was tuned to minimize the average MSE over all episodes. Although the Kalman gain can be slow to converge in some cases (Fig. A performance quickly close to optimal (Fig. B).
Fig. B shows the achievable optimal control cost as function of delay for four different controllers: the optimal linear cawaar-uuuuui---e-ee -aaussian (LQG) controller that uses time-dependent gains, the biologically implausible ANN (cf. Supplementary Fig. SIC) that uses time-invariant gains, mroee-frred
Note that LQG uses the model for state inference as well as for control. Our Bio-OFC uses the model only to infer the latent state, but uses model-fre controller. In contrast, model-based reinforcement learning algorithms typically assume knowledge of the current state and the model only for control.

---

A
B
1.0
0.25
Filter Gain
0.8
0.20
0.6
MSE
0.15
0.4
0.10
0.2
0.05
0.0
2000
4000 6000 Episodes
8000 10000
2000
4000 6000 8000 10000 Episodes
Figure 2: Bio-OFC adapts to changing noise statistics. (A) Filter gain (colors denote different elements of the gain matrix) and (B) mean squared prediction error (MSE) in the simple LQG task with 2 latent dimensions and 2-d observations (LDS1, see text for details). Solid lines show the mean over 20 runs. Dashed lines indicate the optimal filter gain and the corresponding average MSE After 2500 episodes the observation noise covariance W decreased to diag .01), after 5000 to diag(.01. 01), and after 7500 back tiag(.04. .25).
A
B
11-
ANN
LQU
11
ANN
LDG
Bio OFC
Bo OFC
10
Modelk/we
10
Model-fre
9-
Cost
9
Cost
8
7
10
5
6-
5-
5
10
2
6
8
10
2
8 10
Delay
Delay
Figure 3: Optimal cost of Bio-OFC is close to that of LQG for various delays. Optimal cost as function measurement delay obtained by different methods. We used the simple LQG tasks with 2 latent dimensions and (A) 2-d observations (LDS1, see text for details) and (B) 3-d observations (LDS2). LQG uses the optimal time-dependent filter and feedback gains, ANN uses the network of Supplementary Fig. (STC. Bio-OFC the network of Fig. [ (cf. Supplementary Figs. [STB and S2)), and model-free directly maps from noisy delayed observations to control, u, K y:-r (cf. Supplementary Fig. E)... Shown is the mean cost + SEM over 10000 episodes.
using policy gradient method applied directly observations (inset), and Bio-OFC that update the current state estimate &; directly using the delayed measurement y:-r (cf. Eq. ( and Fig. [ For the sake of biological plausibility, we imposed time-invariant gains and direct state estimate updates based on delayed measurement. These results clearly demonstrate that Bio-OFC is robust to sensory delays. Specifically, while it is expected that Bio-OFC will not learn solution quite as good LQG, our results show that the solution found by it is not very far off. In general, a model can be useful in two ways: It facilitates filtered estimate that is useful even in the absence of measurement delays (see LQG vs model-frre free for delay in Fig. 3), and in the presence delays the model helps to bridge the gap by predicting forward in time.
Full system identification. We next considered the case of system identification, i.e. learning the weight matrices A, B, C' in addition to L, using Eqs. small random numbers drawn from (0, 0.01). For LDSI, and L were initialized as diagonal dominated random matrices, with diagonal elements drawn uniformly randomly from [0.5, 1 and off-diagonal ones from [0, 0.5). For LDS2, they were drawn from (0, 0.01) under the constraint that the symmetric part of LC has positive eigenvalues. Controls u, were drawn from (0, 0.25). Fig. AA show how the mean squared prediction error converges to the optimal values from Fig, 3 (dashed horizontal lines). We also considered Bio-OFC that learns in environment LDS2, but
16 We initialized A and B with [6]

---

over-represent' the latent space, assuming it is 3-d instead of 2-d. The square matrices C' and L were initialized analogously LDSI. Fig. [ shows that this over-representation does not affect performance.
B
C
Del
Delay
Delay
1.5
1.0
0.5
0.5
0.5
0.0
0.0
1000
4000
5000
1000
4000
5000
1000
3000
Episodes
Episodes
Episodes
F
40-
Delay
40-
Delay
40
Delay
30
30
30
20
20-
20
10
10
Episodes 6000
2000
6000
2000
Episodes
Episodes
Figure 4: Open-loop Bio-OFC converges to optimally achievable MSE and cost given the delay. The solid (shaded) curve depicts the mean (+SEM) of 20 runs with different random initial weights, smoothed using running median with window size 51. The dashed horizontal lines show the asymptotic values from Fig. 3 The dashed vertical line depicts the time when learning was stopped and the exploratory stochasticity in the controller removed. Mean squared prediction error during system identification, using plasticity rules Eqs. L3 16) for the filter and random Gaussian controller input, as function of episodes for (A) 2-d observations (LDS1), (B) 3-d observations (LDS2) and (C) 3-d observations (LDS2) with over-representing Bio-OFC that assumes 3 instead of the actual latent dimensions. Cost as function of episodes, using plasticity rules Eqs. (21 for the controller, for (D) 2-d observations (LDSI, see text for details), (E) 3-d observations (LDS2) and (F) 3-d observations (LDS2) with an over-representing Bio-OFC that assumes 3 instead of the actual 2 latent dimensions.
System identification followed by control. After performing system identification for 5000
episodes, we kept A, B, C, L fixed and transitioned to learning the controller K using Eqs. (21 23
for another 5000 episodes with controller noise & ~ NO 0.04), ef. Eq. (20) and Fig. D- The stochasticity in the controller results in an excess cost. Using deterministic controller ( 0) for another 1000 episodes reveals that the network converged the optimal cost from Fig. 3 (dashed horizontal lines). We used two learning rates, one for A, B, C, L and one for K with momentum m = 0.99 for the latter. Learning rates that quickly yield good final performance were chosen by minimizing the sum of average reward during and after learning using Optuna (29], a hyperparameter optimization framework freely available under the MIT license. Different noise levels a and momenta m are considered in Supplementary Figs. an SanndS7reeee respectiiv..
Simultaneous system ID and control. Performing system identification prior to learning a con- troller known open-loop adaptive control and common approach in control theory. Recent advances in the control community tackle the more challenging problem of closed-loop control [4] simultaneously identifying both while controlling the system. Our network is capable of closed-loop control with no separate phases for system identification and control optimization necessary. In contrast to our work, the only other proposed neural implementation that also includes control 6 requires separate phases for system-II) and control. Fig. 5 shows how the control cost evolves in time when the weights are updated according to Eqs. 13 16 and Eqs. (21 while using the controller
designed inputs of Eq. (20). Again, using a deterministic controller (E 0) for another 1000 episodes
WWee a controller designs the inputs based on the history of inputs and observations. the inputs become highly correlated with the past process noise sequences, which prevents consistent and reliable parameter estimation with standard system identification techniques.
8

---

B
Delay
40
Delay
40
Delay
30
30
30
10
10-
10
2000
8000
10000
4000 6000 10000
8000 10000
Episodes
Figure 5: Closed-loop Bio-OFC converges to optimally achievable cost given the delay. The solid (shaded) curve depicts the mean (+SEM) of 20 runs with different random initial weights, smoothed using running median with window size The dashed horizontal lines show the asymptotic values from Fig. 3 The dashed vertical line depicts the time when learning was stopped and the exploratory stochasticity in the controller removed. Cost as function of episodes, using plasticity rules Eqs. [3 for the filter and Eqs. 21 23 for the controller simultancously, for (A) 2-d observations (LDSI, see text for details), (B) 3-d observations (LDS2) and (C) 3-d observations (LDS2) with an over-representing OiO that assumes instead of the actual latent dimensions.
trials 1-250
251-500
501 751-750
751-1000
Figure 6: Bio-OFC captures human performance when learning to adapt to force field. (A) Model trajectories during training. Performance plotted during the first, second, third, and final 250 targets. Dots show the mean and are 10 ms apart, shaded area shows kernel density estimate thresholded at 0.04. (B) Averages of human hand trajectories during training [I] Copyright 01994 Society for Neuroscience.
reveals that the network converged to the optimal cost. Different noise levels a and momenta m are considered in Supplementary Figs. 58 and S9 respectively.
4.2
Reaching movements
To connect back to a biological sensory-motor control task, we considered the task of making reaching movements the presence externally imposed forces from mechanical environment [ (Supplementary Material). We initialized the weights of our network to the values that are optimal null foree field, using unit time of 10 and a sensory delay of ms (i.e. 5 5), has been measured experimentally B0 Bio-OFC successfully captures the characteristics of human trajectories in the null field as well as the force field, cf. Fig. (SOO, Bio-OFC adapts the force field by updating its weights according to plasticity rules Eqs. [B 16 for the filter and Eqs. (31 for controller. Figs. annddSS show that this captures human performance during the training period. Switching off learning in the controller yields virtually identical results, ef. Fig. S12) thus learning is driven primarily by changes in the estimator. Using signal-dependent motor noise the plant 31
which increases with the magnitude of the control signal, also yiclds similar results, ef. Figs. S13 S14

---

4.3 Simplified winged flight
32 environment which simulates winged flight in 2-d with simplified dynamics (cf. Fig. S115. Here, the agent controls the flapping frequency of each wing individually, producing an impulse up and away from the wing (i.e. direction up and left when flapping the right wing). The agent receives sensory stimuli which are delayed by 100 ms (equivalent
For our final example, we designed an OpenAI gym
time-steps of the simulation). The goal of the agent is to fly stabilize itself against gravity, the environment wind, and stochastic noise in the control system. The agent suffers cost that is proportional to the L1 distance to the target, and the L.I magnitude of the control variables. This 1.1 cost was chosen to verify the flexibility of our algorithm when the cost deviates from the assumptions of LQR, where the cost is quadratic. We compare the performance of Bio-OFC to policy gradient. We find that, because of the delay, the agent trained with policy gradient overshoots the target and needs to backtrack, cf. Fig. [T However, the agent trained with Bio-OFC flies directly towards the target with significant overshoot, ef. Fig. (Sgb] For more details and video demonstration (gym-fly-den dem.ooooo....... . mp4) see the Supplementary Material.
fixed target and
5
Discussion
In this work, we developed a biologically plausible neural algorithm for sensory-motor control that learns a model of the world, and uses the ability to preview future, through a neurally plausible version of the Kalman filter, learn an appropriate control policy. This neural circuit has the capacity to build an adequate representation of the appropriate state space, can deal with sensory delays, and actively explores the action space to execute appropriate control strategies.
We used a model-free controller, primarily due to its simplicity and biological plausibility. A model- based controller would need access to the model, i.e. weight matrices A and B, which can result in a weight transport problem. However, model-based control has advantages such as higher sample efficiency and the ability be transferable other goals and tasks. An interesting question for future work would be how to combine state estimation via the Kalman filter with model-based control in a biologically plausible manner.
One limitation of this work is that it is in the framework of linear control theory. Locally linearized dynamics [33] has been suggested to generalize the Kalman filter. The inputs could also be processed using additional neural network layers to obtain a representation that renders the dynamics linear 34. In several normative approaches towards neurally plausible representation learning, simply constraining neural activity to be nonnegative while retaining the same objective functions, allowed one to move from, say, PCA [35] clusters (36] and manifolds [37]. Our work could be the starting point of a similar generalization.
We considered uniform delay for all stimuli. In the case of motor control, proprioceptive feedback is faster than visual feedback [30]. Our model readily extends to the case of various, but known, delays for differrnt modalities. The prediction error coding neurons merely need to combine predictions and measurements adequately, i.e. the synaptic delay associated with prediction has to match with the sensory delay for that modality. We believe learning the appropriate delay could be implemented by extending the state space using lag vectors 2 which we leave for future work.
In line with overall brain architecture [38] and the predictive coding framework 10 our model suggests the brain employs a recurrent network to generate predictions and actions by constantly attempting to match incoming sensory inputs with top-down predictions [1]. Our model can also be mapped to brain regions putatively contributing to optimal feedback control [30 39]. Specifically,
has been proposed that the cerebellum performs system identification, parietal cortex performs state
estimation, and primary and premotor cortices implement the optimal control policy by transforming state estimates into motor commands. Also, basal ganglia may be involved in processing cost/reward
39
This work proposed a concrete neural architecture that takes up the challenge of online control in a changing world [41,, with delay, and using biologically plausible synaptic update rules. The learning algorithm performs well for several tasks, even with some drastic approximations. Future exploration of its limitations would provide further insights into the general nature of biologically constrained control systems.
10

---

Acknowledgments and Disclosure of Funding
AS thanks A. Acharya, S. Saha and S. Sridharan for discussions. JF, SG, SF and DC were internally funded by the Simons Foundation. AS was partly funded by Simons Foundation Neuroscience grant SF 626323 during this work.
References
[1] R. Shadmehr and F. A. Mussa-Ivaldi. Adaptive representation of dynamics during learning of a motor task. Journal of Neuroscience, 14(5):3208-33 -2204,,22,,42 1994.
[2] E. Todorov and M. I. Jordan. Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5(11):1226 122111226,,,,2 2002.
[3] S. Scott. Optimal feedback control and the neural basis of volitional motor control. Nature Reviews Neuroscience, 5(7):532-54) -.-55552555.22 2004.
[4] S. Deneve, J.RR. Duhamel, and A. Pouget. Optimal sensorimotor integration in recurrent cortical networks: neural implementation of Kalman filters. Journal of Neuroscience, 27(21):5744 5756, 2007.
[5] R. Wilson and L.. Finkel. A neural implementation of the Kalman filter. In Advances in Neural Information Processing Systems, volume 22, pages 2062-2002777 2009.
[6] R. Linsker. Neural network learning of optimal Kalman prediction and control. Neural Networks, 21(9):1328-1343 2008.
[7] A. Kutschireiter. S. C. Surace, H. Sprekeler, and J.-P. Pfister. Nonlinear Bayesian filtering and learning: a neuronal dynamics for perception. Scientific Reports, 701):1-13, 2017.
[8] B. Millidge, A. Tschantz, Seth, and Buckley. Neural Kalman filtering. urXiv:2102.10021, 2021.
[9] D. McNamee and D. M. Wolpert. Internal models in biological control. Annual Review of Control. Robotics, and Autonomous Systems, 2:339-364, 2019.
[10] R. P. Rao and D. H. Ballard, Predictive coding in the visual cortex: a functional interpretation of some extra- extra-classical receptive-fide field effects. Nature Neuroscience, 2(1):79-87, 1999.
[11] A. Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3):181-20) 204, 2013.
[12] R. J. Williams. Simple statistical saadientfoollowigg algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-22 1992.
[13] H. S. Seung. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 40(6):1063-1073, 2003.
[14] S. Lale, K. Azizzadenesheli, B. Hassibi, and A. Anandkumar. Logarithmic regret bound in partially observable linear dynamical systems. In Advances in Neural Information Processing Systems, volume 33, pages 19876-20883 2020.
[15] B. Lee and A. Lamperski. Non-asymptotic closed-loop system identification using autoregressive processes and hankel model reduction. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 3419-3424, 2020.
[16] R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35-45 1960.
[17] A. Rajeswaran, K. Lowrey, F. V. Todorov, and M. Kakade. Towards generalization and simplicity in continuous control. In Advances in Neural Information Processing Systems, volume 30, 2017.
[18] H. Alexander. State estimation for distributed systems with sensing delay. In Data Structures and Target Classification, volume 1470, pages 103-111. International Society for Optics and Photonics, 1991.
[19] D. Larsen, N. A. Andersen, O. Ravn, and N. K. Poulsen. Incorporation of time delayed measurements in discrete-time kalman filter. In Proceedings of the 37th IEEE Conference Decision and Control, volume 4, pages 3972-3977. IEFF, 1998.
[20] M. Verhaegen and V. Verdult. Filtering and system identification: least squares approach. Cambridge University Press, 2007.
[21] J. Friedrich and M. Lengyel. Goal-directed decision making with spiking neurons. Journal of Neuroscience, 36(5):1529 1546, 2016.
[22] R. S. Sutton, D. A. McAllester, S. P. Singh, Y. Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 99, pages 1057-1063, 1999.

---

[23] H. Kimura, M. Yamamura, and S. Kobayashi. Reinforcement learning by stochastic hill climbing on discounted reward. In Machine Learning Proceedings 1995, pages 295-303. Elsevier, 1995.
[24] J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 1-319-3500 2001.
[25] Fazel, Ge, Kakade, and M. Mesbahi. Global convergence of policy gradient methods linear quadratic regulator. In International Conference on Machine Learning, pages 1467-1476. PMLR, 2018.
[26] R. L. Redondo and R. G. Morris. Making memories last: the synaptic tagging and capture hypothesis. Nature Reviews Neuroscience, 12(1):17-30, 201
[27] J. Friedrich, R. Urbanezik. and W. Semn. nemporl credit assignment in neuronal population learning. PLoS Computational Biology. 1002092, 2011.
[28] B. Recht, tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems, 2:253-279, 2019.
[29] T. Akiba, Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: next-generation hyperparameter opti- mization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
[30] S. H. Scott. The computational and neural hasis of voluntary motor control and planning. Trends in Cognitive Sciences, 16(11):541-55 2012.
[31] C. M. Harris and D. M. Wolpert. Signal-dependent noise determines motor planning. Nature, 394(6995),800-78, 1998.
[32] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym, 2016.
[33] S. J. Julier and J. K. Uhlmann. New extension Kalman filter to nonlinear systems. Signal processing, sensor fusion, and target recognition VI, volume 3068, pages 182-193, International Society for Optics and Photonics, 1997.
[34] B. Lusch, Kutz, and L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9(1):1-10, 2018.
[35] C. Pehlevan, T. Hu, and D. B. Chklovskii. hebbian/anti hataan/bbnnn-ebbiann neural network for linear subspace Icarning: A derivation from multidimensional scaling of streaming data. Neural Computation, 2767):1461- 1495, 2015.
[36] C. Pehlevan and D. B. Chklovskii. A normative theory of adaptive dimensionality reduction in neural networks. In Advances in Neural Information Processing Systems, volume 28, pages 2269-2277, 2015.
[37] A. M. Sengupta, C. Pehlevan, M. Tepper, A. Genkin, and D. B. Chklovskii. Manifold-tiling localized receptive fields are optimal in similarity neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
[38] H. Sohn, N. Meirhaeghe, R. Rajalingham, and M. Jazayeri. A network perspective on sensorimotor learning. Trends in Neurosciences, 2020.
[39] R. Shadmehr and W. Krakauer. computational neuroanatomy for motor control. Experimental Brain Research, 185(3):359-33) 381, 2008.
[40] D. I. Todorov, R. A. Capps, W. H. Barnett, E. M. Latash, T. Kim, K. C. Hamade, S. N. Markin, I. A. Rybak, and Y. I. Molkov. The interplay between cerebellum and basal ganglia in motor adaptation: modeling study. PLoS ONE, 14(4)xc0214926, 2019.
[41] N. Agarwal, E. Hazan, A. Majumdar, and K. Singh. A regret minimization approach to iterative learning control. In International Conference on Machine Learning, volume 139, pages 100-109. PMLR, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section Conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See Supple- mentary Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section Experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Figs. 3 show mean + SEM
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Supplementary Material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See reference 29
(b) Did you mention the license of the assets? [Yes] See Section Experiments.
(c) Did you include any assets either in the supplemental material as URL? [N/A]
No new assets were produced.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "xmX1595CBm", "reviewer": "Reviewer_yCNb", "summary": "The authors propose a model of optimal feedback control for linear dynamical systems that can be realized via biologically plausible learning rules. The authors recognize the importance of integrating both prediction and control and achieve this by integrating two modules -- a model-based state estimation module that combines noisy measurements with internally generated predictions, and a model-free controller module that selects actions based on the estimated state. They demonstrate that the state estimator module can learn the parameters of the system using Hebbian-like weight updates provided the model parameters are initialized appropriately. They also demonstrate that the effect of sensory delay on the proposed model is comparable to what would be expected from an optimal model. The control module uses a standard policy-gradient method to learn a mapping from states to actions. The authors also evaluate their model in a neuroscience-inspired reaching task as well as a new flying task designed in OpenAI gym.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is fairly original in its goal of performing system identification using biologically plausible algorithms and integrating it with a model-free controller. The authors have clearly identified how their contribution builds on previous work. The mathematical proofs and techniques are sound but the results fall short of adequately highlighting the consequences of key assumptions and choices proposed in this work. Although biological plausibility is a huge constraint, I find the focus on linear dynamical systems somewhat restrictive particularly since the problems faced by the brain are almost always nonlinear. The authors certainly acknowledge this limitation but they can be more elaborate in pointing the way for readers interested in taking these insights to nonlinear settings. Here are some specific questions and suggestions.\n\n(1) I found the idea of simply replacing $C^\\text{T}$ with $L$ really cool. But what are the precise consequences? How would the results shown in Figure 2 look if you used $C^\\text{T}$ ? \n\n(2) How does the learning rate and asymptotic performance depend on the degree of initial alignment between $C^\\text{T}$ and $L$ ? In general, the authors should better highlight the interaction between algorithmic assumptions and performance.\n\n(3) The authors of ref [8] stated in their study that parameter $C$ could not be learned with Hebbian-like rule. Why does it work here? Was it just a matter of using the right initialization or were there other reasons? \n\n(4) **Supplementary Lines 91-92**: *The marginal mean-squared error decreases if the angle between the gradient gθ for parameter θ and\n92 the update ∆θ is less than 90◦, i.e. gθ⊤∆θ > 0.* This certainly explains why the MSE would decrease **initially**. Is there a guarantee that this condition is met **throughout** learning when $C$ and $L$ are learned simultaneously according to the proposed learning rule?\n\n(5) **Figure 3**: Why is model-based state estimation still better than model-free even when delay is 10, given that the trial length is only 10?\n\n(6) **Lines 180-183**: *It shows that a model can be useful in two ways: It facilitates a filtered estimate that is useful even in the absence of measurement delays (see LQG vs model-free for delay 0 in Fig. 3), and in the presence of delays the model helps to bridge the gap by predicting forward in time.* These statements are a tad underwhelming and are true in general, independent of this work. Could you highlight the specific consequences of the approximations made in the proposed model by comparing Bio-OFC against LQG? I could be mistaken, but I did not see any new features in the proposed model that make it robust to sensory delays.\n\n(7) **Lines 208-210**: *Strikingly, our network is capable of closed-loop control with no separate phases for system identification and control optimization necessary*. While it is reassuring to know that the state estimator and the controller could be learned in parallel, why is this unexpected?\n\n(8) **Reaching task (Figure S8)**: Although the task is biologically motivated, it is greatly simplified by assuming that the motor command is already in cartesian coordinates as opposed to controlling the torques applied on joints (which would make the system nonlinear). This is fine, but it means that the correspondence to human behavior is hard to interpret without additional assumptions. Questions: Was there a separate estimator/controller for each reach condition? Was the learning in response to force field driven primarily by changes in the estimator or controller? How does the model learn a time-dependent A?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(1) **Lines 256-258**: *Locally linearized dynamics [33] has been suggested to generalize the Kalman filter. The inputs could also be processed using additional neural network layers to obtain a representation that renders the dynamics linear.*  Could you be a bit more specific about how the ideas proposed here would generalize to nonlinear systems while still retaining the biological plausibility?\n\n(2) **Lines 276-278**: *\"Our model suggests that system identification is performed by adapting connections between cerebellum and parietal cortex, actions are refined by changes in synapses connecting parietal cortex with premotor cortices\".* This is a particularly strong speculation because, to my knowledge, connections from parietal to premotor are not thought to implement model-free computations. The state estimation / controller modules better correspond to cortical / subcortical areas. Which brings me to another question. Perhaps you could also discuss the limitations of using a purely model-free controller?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "eoz-aIYQYFJ", "reviewer": "Reviewer_pVEa", "summary": "The paper introduces a biologically plausible neural implementation of optimal feedback control, which considers that sensory feedback is delayed, and combines adaptive Kalman filtering and model free control. The proposed algorithm can be applied closed-loop, i.e., the algorithm can be used for online control such as natural neural networks, and does not need additional training and calibration phases. The performance of the method has been assessed in several simulation experiments. Simulation results confirm that appropriate sensory-motor control can be achieved.\n\nThe main contribution is the proposed network named Bio-OFC, which does not require information about noise and system dynamics and allows closed-loop control.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is novel, timely, clearly written, well organised and well presented. The main contributions are clearly highlighted and motivated. The authors reasoning and rationales, and the use of methods is sound. Limitations have been discussed; code and supplemental material is provided. This is an interesting approach and useful feedback-based learning approach. \n\nThe work addresses a difficult task in a new way. The claims are supported by simulation studies. Computational complexity and run-times of the method are not discussed. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Major limitation has been addressed. Discussion of the impact on society is not yet relevant for this work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns. Simulation and synthetic data were used to assess the performance of the system.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "UrAey-MJM6Q", "reviewer": "Reviewer_s8QW", "summary": "The authors propose a neural network that learns state estimation and control of a linear system via bio-plausible plasticity rules that implement adaptive Kalman filtering and policy gradient. Their scheme incorporates sensory delays, does not need to know noise covariances and can operate in closed loop without separate learning and execution phases.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper lucidly introduces neural optimal control, reviews the literature widely, and implements an adaptive neural Kalman filter and controller with bio-plausible learning rules. Their system can handle delays (known a priori) and noise covariances (not known a priori). A key insight in obtaining local learning rules for the Kalman filter is to replace $C^T$ with $L$ albeit requiring prior initialization constraints. The authors learn the controller by a form of REINFORCE - output node perturbation, made bio-plausible using eligibility traces. The authors test their network to control example linear systems, and demonstrate a few advantages of their method -- here, they should clarify why these advantages ensue. While currently limited to linear systems, the authors suggests ways in which this limitation may be overcome. Overall, this implementation simultaneously addresses multiple issues in bio-plausible adaptive neural control, and is clearly exposited. I expect it to enable further cross-fertilization between the control and neural reinforcement learning communities.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DyP-4mvTIbB", "reviewer": "Reviewer_MZDW", "summary": "The paper proposes a method (bio-OFC) for learning optimal feedback control policies in a neural network with local learning rules. The method builds on the dominant model of biological motor control, optimal feedback control (Todorov and Jordan, 2002), which, given knowledge of system dynamics and noise covariances, is able to account well for variance patterns in human motor behaviour. A major limitation of OFC as a model of biological motor systems is that it does not learn the system dynamics and previously has not clearly mapped on to biologically-plausible implementations. \nIn this work, the authors propose a method for adapting the OFC framework so that it instead learns a forward predictive model of the system dynamics, and uses policy gradients to learn control. Unlike the original OFC formulation, this method doesn’t require a priori knowledge of noise covariances. The authors perform experiments in three environments, a simple control task (discrete time double integrator), a classic psychophysical reaching task, and a ‘winged flight’ control task. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the work provides a development on a previous model with similar aspirations to provide a neural implementation of OFC (Linkser, 2008), which although acknowledged in Table 1 with a checklist comparison of common vs different features to their own work, was unfortunately not simulated under the same tasks for a more in-depth comparison. This paper did however test their method more thoroughly than Linkser 2008 and on new tasks, which is valuable.\nThe authors claim that one of the novel contributions of their method is that it functions under delayed sensory feedback, however the original OFC implementation by Todorov and Jordan did consider the case of delayed sensory feedback. It is possible that learning under these delayed sensory conditions is more difficult to handle than in the 2002 control-only formulation, in which case the authors should clarify why this is so.\nThis paper compares the control cost performance of their method to optimal LQG control, a NN controller that is ‘biologically implausible’, and a controller which does not learn a predictive dynamics model and instead maps delayed sensory observations linearly to control output (Figure 3). The authors’ method appears to perform fairly well (Fig3, 4) as although it is expected that bio-OFC will not learn a solution quite as good as LQG control, its not very far off, and bio-OFC performs only a little bit worse than the ANN with weight copies (blue). \nOverall the results section was a little hard to follow, I think because it wasn't immediately clear what was learned from the results from each experiment or why each experiment was run. Some important results were also relegated to Supplemental information, e.g. results from the reaching movement experiment, which would have been valuable to include in the main text, seeing as the link to biological motor control seems to be a major focus of this paper.\nA way to strengthen the connection to biological motor control would be to run experiments in which there is signal-dependent motor noise in the plant (Harris & Wolpert, 1998), and compare behaviour to humans.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper had uniform support from the reviewers, and is a clear accept. To improve the work, I highly suggest you:
1 . Add the additional experiment requested by a reviewer
2. More clearly outline the derivation for the update equations (we discussed this in our discussion and concluded that they were correct, but it was not immediately obvious just from the text)
3. Incorporate other suggestions by the reviewers.
Decision: Accept (Spotlight)