Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a neural network that learns state estimation and control of a linear system via bio-plausible plasticity rules that implement adaptive Kalman filtering and policy gradient. Their scheme incorporates sensory delays, does not need to know noise covariances and can operate in closed loop without separate learning and execution phases.
main_review: This paper lucidly introduces neural optimal control, reviews the literature widely, and implements an adaptive neural Kalman filter and controller with bio-plausible learning rules. Their system can handle delays (known a priori) and noise covariances (not known a priori). A key insight in obtaining local learning rules for the Kalman filter is to replace $C^T$ with $L$ albeit requiring prior initialization constraints. The authors learn the controller by a form of REINFORCE - output node perturbation, made bio-plausible using eligibility traces. The authors test their network to control example linear systems, and demonstrate a few advantages of their method -- here, they should clarify why these advantages ensue. While currently limited to linear systems, the authors suggests ways in which this limitation may be overcome. Overall, this implementation simultaneously addresses multiple issues in bio-plausible adaptive neural control, and is clearly exposited. I expect it to enable further cross-fertilization between the control and neural reinforcement learning communities.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a model of optimal feedback control for linear dynamical systems that can be realized via biologically plausible learning rules. The authors recognize the importance of integrating both prediction and control and achieve this by integrating two modules -- a model-based state estimation module that combines noisy measurements with internally generated predictions, and a model-free controller module that selects actions based on the estimated state. They demonstrate that the state estimator module can learn the parameters of the system using Hebbian-like weight updates provided the model parameters are initialized appropriately. They also demonstrate that the effect of sensory delay on the proposed model is comparable to what would be expected from an optimal model. The control module uses a standard policy-gradient method to learn a mapping from states to actions. The authors also evaluate their model in a neuroscience-inspired reaching task as well as a new flying task designed in OpenAI gym.
main_review: The work is fairly original in its goal of performing system identification using biologically plausible algorithms and integrating it with a model-free controller. The authors have clearly identified how their contribution builds on previous work. The mathematical proofs and techniques are sound but the results fall short of adequately highlighting the consequences of key assumptions and choices proposed in this work. Although biological plausibility is a huge constraint, I find the focus on linear dynamical systems somewhat restrictive particularly since the problems faced by the brain are almost always nonlinear. The authors certainly acknowledge this limitation but they can be more elaborate in pointing the way for readers interested in taking these insights to nonlinear settings. Here are some specific questions and suggestions.

(1) I found the idea of simply replacing $C^\text{T}$ with $L$ really cool. But what are the precise consequences? How would the results shown in Figure 2 look if you used $C^\text{T}$ ? 

(2) How does the learning rate and asymptotic performance depend on the degree of initial alignment between $C^\text{T}$ and $L$ ? In general, the authors should better highlight the interaction between algorithmic assumptions and performance.

(3) The authors of ref [8] stated in their study that parameter $C$ could not be learned with Hebbian-like rule. Why does it work here? Was it just a matter of using the right initialization or were there other reasons? 

(4) **Supplementary Lines 91-92**: *The marginal mean-squared error decreases if the angle between the gradient gθ for parameter θ and
92 the update ∆θ is less than 90◦, i.e. gθ⊤∆θ > 0.* This certainly explains why the MSE would decrease **initially**. Is there a guarantee that this condition is met **throughout** learning when $C$ and $L$ are learned simultaneously according to the proposed learning rule?

(5) **Figure 3**: Why is model-based state estimation still better than model-free even when delay is 10, given that the trial length is only 10?

(6) **Lines 180-183**: *It shows that a model can be useful in two ways: It facilitates a filtered estimate that is useful even in the absence of measurement delays (see LQG vs model-free for delay 0 in Fig. 3), and in the presence of delays the model helps to bridge the gap by predicting forward in time.* These statements are a tad underwhelming and are true in general, independent of this work. Could you highlight the specific consequences of the approximations made in the proposed model by comparing Bio-OFC against LQG? I could be mistaken, but I did not see any new features in the proposed model that make it robust to sensory delays.

(7) **Lines 208-210**: *Strikingly, our network is capable of closed-loop control with no separate phases for system identification and control optimization necessary*. While it is reassuring to know that the state estimator and the controller could be learned in parallel, why is this unexpected?

(8) **Reaching task (Figure S8)**: Although the task is biologically motivated, it is greatly simplified by assuming that the motor command is already in cartesian coordinates as opposed to controlling the torques applied on joints (which would make the system nonlinear). This is fine, but it means that the correspondence to human behavior is hard to interpret without additional assumptions. Questions: Was there a separate estimator/controller for each reach condition? Was the learning in response to force field driven primarily by changes in the estimator or controller? How does the model learn a time-dependent A?
limitations_and_societal_impact: (1) **Lines 256-258**: *Locally linearized dynamics [33] has been suggested to generalize the Kalman filter. The inputs could also be processed using additional neural network layers to obtain a representation that renders the dynamics linear.*  Could you be a bit more specific about how the ideas proposed here would generalize to nonlinear systems while still retaining the biological plausibility?

(2) **Lines 276-278**: *"Our model suggests that system identification is performed by adapting connections between cerebellum and parietal cortex, actions are refined by changes in synapses connecting parietal cortex with premotor cortices".* This is a particularly strong speculation because, to my knowledge, connections from parietal to premotor are not thought to implement model-free computations. The state estimation / controller modules better correspond to cortical / subcortical areas. Which brings me to another question. Perhaps you could also discuss the limitations of using a purely model-free controller?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper introduces a biologically plausible neural implementation of optimal feedback control, which considers that sensory feedback is delayed, and combines adaptive Kalman filtering and model free control. The proposed algorithm can be applied closed-loop, i.e., the algorithm can be used for online control such as natural neural networks, and does not need additional training and calibration phases. The performance of the method has been assessed in several simulation experiments. Simulation results confirm that appropriate sensory-motor control can be achieved.

The main contribution is the proposed network named Bio-OFC, which does not require information about noise and system dynamics and allows closed-loop control.
main_review: The work is novel, timely, clearly written, well organised and well presented. The main contributions are clearly highlighted and motivated. The authors reasoning and rationales, and the use of methods is sound. Limitations have been discussed; code and supplemental material is provided. This is an interesting approach and useful feedback-based learning approach. 

The work addresses a difficult task in a new way. The claims are supported by simulation studies. Computational complexity and run-times of the method are not discussed. 

limitations_and_societal_impact: Major limitation has been addressed. Discussion of the impact on society is not yet relevant for this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method (bio-OFC) for learning optimal feedback control policies in a neural network with local learning rules. The method builds on the dominant model of biological motor control, optimal feedback control (Todorov and Jordan, 2002), which, given knowledge of system dynamics and noise covariances, is able to account well for variance patterns in human motor behaviour. A major limitation of OFC as a model of biological motor systems is that it does not learn the system dynamics and previously has not clearly mapped on to biologically-plausible implementations. 
In this work, the authors propose a method for adapting the OFC framework so that it instead learns a forward predictive model of the system dynamics, and uses policy gradients to learn control. Unlike the original OFC formulation, this method doesn’t require a priori knowledge of noise covariances. The authors perform experiments in three environments, a simple control task (discrete time double integrator), a classic psychophysical reaching task, and a ‘winged flight’ control task. 

main_review: Overall the work provides a development on a previous model with similar aspirations to provide a neural implementation of OFC (Linkser, 2008), which although acknowledged in Table 1 with a checklist comparison of common vs different features to their own work, was unfortunately not simulated under the same tasks for a more in-depth comparison. This paper did however test their method more thoroughly than Linkser 2008 and on new tasks, which is valuable.
The authors claim that one of the novel contributions of their method is that it functions under delayed sensory feedback, however the original OFC implementation by Todorov and Jordan did consider the case of delayed sensory feedback. It is possible that learning under these delayed sensory conditions is more difficult to handle than in the 2002 control-only formulation, in which case the authors should clarify why this is so.
This paper compares the control cost performance of their method to optimal LQG control, a NN controller that is ‘biologically implausible’, and a controller which does not learn a predictive dynamics model and instead maps delayed sensory observations linearly to control output (Figure 3). The authors’ method appears to perform fairly well (Fig3, 4) as although it is expected that bio-OFC will not learn a solution quite as good as LQG control, its not very far off, and bio-OFC performs only a little bit worse than the ANN with weight copies (blue). 
Overall the results section was a little hard to follow, I think because it wasn't immediately clear what was learned from the results from each experiment or why each experiment was run. Some important results were also relegated to Supplemental information, e.g. results from the reaching movement experiment, which would have been valuable to include in the main text, seeing as the link to biological motor control seems to be a major focus of this paper.
A way to strengthen the connection to biological motor control would be to run experiments in which there is signal-dependent motor noise in the plant (Harris & Wolpert, 1998), and compare behaviour to humans.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper had uniform support from the reviewers, and is a clear accept. To improve the work, I highly suggest you:
1 . Add the additional experiment requested by a reviewer
2. More clearly outline the derivation for the update equations (we discussed this in our discussion and concluded that they were correct, but it was not immediately obvious just from the text)
3. Incorporate other suggestions by the reviewers.
Decision: Accept (Spotlight)