Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new learning rule based on action surprise, which is meant to facilitate off policy reinforcement learning and explain a variety of biological results from DA neuron and basal ganglia recordings, some of which (such as DA coding of novelty and other action related codes) were not thought to be directly linked to reinforcement learning. The authors provide some empirical results comparing the proposed model to other alternatives and show relevant correlates to some biological results, but not direct comparison to behaviour.
strengths_and_weaknesses: I think the proposed idea is indeed very interesting and has great potential to link a variety of seemingly disparate biological results regarding DA and basal ganglia physiology. I also think the authors perform a great literature review and brilliant integration with relevant findings in literature. The clarity of the proposed model is sufficient, although it's a bit abstract at times, e.g. it would be great to see how the mu(s_t) and a_t - mu(s_t) terms would look in practice, as that's the key element of the model and perhaps not sufficiently explained with examples.

My main problem with the paper is that although it seems to clearly target computational cognitive neuroscience (as opposed to machine learning) audience, the type of tasks, training protocols and measurements seem to correspond a lot more to "machine learning" RL, as opposed to what is or could be done in behavioural experiments, whether animal or human. The type of (distance/error dependent) cost function that is presumably used has little in common with that is used in reinforcement learning, where rewards are often scarce and don't provide constant feedback. Training protocols requiring 100k episodes (or anything of that order) are also totally unrealistic in biology, at least if you interpret this as behavioural trials done in learning (and even if you try to relate it to replay of sequences and similar data, 100k is still quite a stretch). There is no behavioural data provided for these 2 tasks that could suggest based on what timeline or mechanisms animals actually learn in these cases, and comparisons with other models (particularly examples, like in Fig.2b given without much explanation of why this happens) are not very informative. While I don't doubt that authors' model can perform well as asserted and their idea (if not necessarily the implementation) may be a great way to reconcile some disparately looking biological data, I think the effort to link model performance to actual experiments is insufficient. It's true that there is not a great deal of data of direct relevance to off-policy RL in behavioural neuroscience, but I doubt that nothing could be found which could be compared more directly.
questions: To improve the paper, the authors could do the following:
- provide more examples about the representation of mu(s_t) and how could it be investigated in neural recordings for example (i.e. is there any specific rather than vague/indirect evidence for such representation)
- address the issue of biological/behavioural relevance of tasks and particularly training methods, as they seem more common in machine learning RL, not behavioural RL (although open field tasks are definitely used widely, but not sure if in exactly the same way), paying particularly care for biological realism of timelines, training protocols and learning rules (e.g. is backprop biologically realistic?..)
- perform direct comparison with behavioural learning data or at the very least discuss how exactly it could be done, or if not, why not.
- perform more meaningful comparisons with other models, if authors feel that such comparisons are needed for the paper (it seems in many cases those other models simply don't work, with only general theoretical explanation provided why not).
As a general comment, I'm quite skeptical of the widespread practice of comparing different models in modelling papers, especially when it's used not as a means to select between several alternatives equally considered by authors, but showcase of the proposed model's superior performance to other existing models, as implementation details of such alternative models may not be chosen with the best intention to make them perform as well as possible, and various creative choices (such as best formalisation of states, actions, rewards) and/or fine tuning may be needed for that purpose, so I think absolute assessment and direct comparison with behavioural or physiological data as opposed to other models are much more useful.

Minor issue: why is there a green line in fig. 3E bottom?
limitations: There was some discussion of limitations, but not really the main limitations/weaknesses of work as I see them (detailed above).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors explains the role of dopamine response at action initiation based on an off-policy actor critic architecture.



strengths_and_weaknesses: This paper makes an important point that the brain's action selection is distributed, so that learning should be off-policy.

The dopamine response at action initiation promotes imitation learning of external controller, which works nice if the external controller is doing a fair job, but can screw up the basal ganglia learning if others are doing badly.

Dopamine response at action initiation had been predicted upon finding an high advantage action in a model-based framework (Doya 1999, Figure 7). This should also be discussed.
Doya K (1999). What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex. Neural Networks, 12, 961-974. https://doi.org/10.1016/S0893-6080(99)00046-5





questions: Dopamine neuron's cue response is quite fast, often preceding action execution. The present framework assumes the use of efference copy of finally selected action somewhere in the brain. Does the timing really match up?

Here is a conceptual question. Action value equation (8) assumes any action that deviates from mu(s) are worse. On the other hand, the advantage equation (12) favors action that deviates from mu(s). Where does this apparent inconsistency come from? Maybe it is a matter of whether you apply (8) to V(s(t)) or V(s(t+1)).

Uchibe proposed an off-policy RL scheme with mixture of controllers, as in the present setting, but uses important sampling, which gives more weights on actions similar to one's own; apparently opposite to what is done by action surprise. Could you comment on the relationship or advantages/weakness?
Uchibe E (2018). Cooperative and Competitive Reinforcement and Imitation Learning for a Mixture of Heterogeneous Learning Modules. Front Neurorobot, 12, 61. https://doi.org/10.3389/fnbot.2018.00061


limitations: The approximation of action value by (8) with fixed sigma would limit the model's applicability.

I updated my score after reading the paper [22].
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: the currently dominant view of RL in the basal ganglia. The authors then show how Q-learning with a particular state-action value function parametrization yields an RL algorithm that is very close to the actor-critic model while being able to learn off-policy. Importantly, this new model has an "action-surprise" term that also allows the model to explain the existence of movement-related modulation in the striatum. The authors show that the model can produce performance advantages. Finally, they discuss how the model might be implemented biologically.
strengths_and_weaknesses: Overall, I really liked this paper. It presented a novel theoretical idea that jointly solves both empirical and computational problems. It was clearly written and compellingly argued.

My only major comment is that I wish the authors had simulated actual neuroscience experiments rather than (or in addition to) their toy setups (the two-joint arm and open field). This feels like a missed opportunity to show how the model can directly explain the relevant empirical data.

Related to the last point, there are experiments that measure dopamine when other action-related areas are manipulated (e.g., inactivated, stimulated, lesioned). It seems like this model should make predictions about these experiments. Similarly, if an animal's actions are influenced in unexpected ways, then the prediction is that this should register in the dopamine signal (though one would have to be careful to control for sensory effects); I'm not aware of studies that look specifically at this question, but it seems like a clear prediction.

Minor comments:

- p. 1: "decreasing amounts movement-modulated dopamine activity" -> "decreasing amounts of movement-modulated dopamine activity"
questions: See comments above.
limitations: The authors don't really address limitations of their approach. I don't see any issues with negative societal impact.
review_title: null

Meta-review: metareview: This paper presents an off-policy RL model for providing biologically plausible explanations of of dopamine activity and basal ganglia recordings.

The reviews express a positive evaluation for the research question and framework and the biological plausibility within the context of the domain application. There are some concerns as to whether these models would compare to more well-established behaviour models. The authors have presented robust clairifications of the main concerns of the reviewers.
confidence: Less certain
Decision: Accept