Temporal-Difference Learning Using Distributed Error Signals
Authors: Jonas Guan
Shon Eduard Verch
Claas A Voelcker
Ethan C Jackson
Nicolas Papernot
William A Cunningham
Venue: NeurIPS 2024 poster (2024)
Keywords: neuroscience
reinforcement learning
temporal-difference learning
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose a new deep Q-learning algorithm that learns to solve RL tasks without sequential propagation of errors, which provides a potential explanation to credit assignment in biological reward-based learning.

Abstract: A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc). Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.

Temporal-Difference Learning Using Distributed Error Signals
Jonas Guan 1,2, Shon Eduard Verch', Claas Voelcker Ethan C. Jackson',
Nicolas Papernot William A. Cunningham 1.3.3
University of Toronto Schwartz Reisman Institute for Technology and Society
Vector Institute
Abstract
A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc) update synaptic weights. Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates learn complex, nonlinear reward-baser based learning tasks. We design a new deep Q-learning algorithm, ARTIFICIAL DOPAMINE, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.
1 Introduction
Computer science and neuroscience have enjoyed longstanding and mutually beneficial relationship. This synergy exemplified by the inception of artificial neural networks, which drew inspiration from biological neural networks. Neuroscience also adopted rmmporal-differena (TD) learning [62] from reinforcement learning (RL) as a framework for biological reward- based learning in the midbrain [23] 59]. At the intersection of these ideas, deep RL has much benefited from and contributed to interdisciplinary progress between the two fields 45
An interesting problem raised in biological learning is how signals transmitted by the neuromodulator dopamine computationally induce coordinated reward- based learning. In the mesolimbic system, dopamine is synthesized by dopamine neurons in the ventral tegmental area (VTA) and transmitted through the mesolimbic pathway to several regions, including the nucleus accumbens (NAc). There, it is synchronously distributed in regionally homogeneous concentrations (58), and serves reward prediction error signal for synaptic adjustments via TD learning (9) 23 Figure shows a conceptual illustration: the medium spiny neurons in the NAc receive region via dopamine. Computationally. however, this theory faces the credit assignment problem [5]: the individual synaptic updates using just local errors must somehow work in coordination to improve ? Are distributed error signals alone sufficient to coordinate neurons to learn complex reward-based learning tasks?
signals distributed locally in their
the collective prediction
NAc dopamine also serves many roles heyond signaling reward prediction emrors 18] its full responsibilitic, are an active area of research. We only focus on its role in error signaling, which is most pertinent to our problem. FFor clarity, this is different from temporal credit assignment problem, oft discussed in RL literature.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Deep RL typically solves the credit assignment problem using backpropagation (BP) [57]. BP propagates the global enror backwards through the network, and computes the gradient (w.t. the global enror) of each layer's s synaptic weights sequentially via the chain rule. In contrast to the synchronously distributed errors in the NAc, BP involves neurons sequentially communicating error signals with each other. This sequential propagation explicitly coor- dinates learning, but also creates dependencies: each layer's updates depend on the error of subsequent layers. This is known as the update locking problem, which is biologically implausible [52), limits parallelization, and cannot explain how distributed signals may support coordinated learning.
Dopamine neuron
Medium spiny neuron
VTA
NAc
Figure 1: Simplified illustration of dopamine distribution in the NAc. Dopamine is syn- thesized in the VTA and transported along axons to the NAc, where picked up by re- ceptors in medium spiny neurons. Dopamine concentrations (error signals) are locally ho- mogenous, but can vary across regions. Con- nections between NAc neurons not shown.
Recent ML research on more biologically plausible alternatives to BP may offer critical insights. PEPITA [1] and Forward-Forwee
(FF) [0 both replace BP's backward learning pass with second forward pass to address update locking. Most relevantly, Hinton [30] made surprising discovery: layers can learn useful representations for subsequent layers even when trained independently of the errors of those subsequent layers. In FF, each layer generates its own prediction and error, and only trained to learn hidden representations that minimize the local error. The subsequent layer takes these representations as input, and achieves better performance over training, despite being unable to send errors the previous layer. This improves the collective global prediction without explicit, sequential coordination of error signals. To the best of our knowledge, these learning principles have not been explored in RL or biological reward- based learning.
Drawing a novel connection, we hypothesize that the computational mechanisms that enable FF's independent, per-layer training may also enable distributed error signals support coordinated reward- based learning. To test our hypothesis, we design ARTIFICIAL. DOPAMINE (AD), a new deep Q- learning algorithm that trains RL agents using only synchronously distributed, per-layer TD errors, and evaluate its performance on a range of discrete and continuous RL tasks. This provides potential ex- planation for credit assignment in NAc dopaminergic learning at the algorithmic level of analysis [42) Our results show that AD can solve many common RL tasks often as well as deep RL algorithms that use backpropagation, despite not propagating error signals between layers. Thus, we computationally demonstrate that distributed errors alone may be sufficient for coordinated soward-based learning.
AD networks inherit several ideas from FF, which differ from traditional neural networks in two significant ways. First, each layer in an AD network computes its own prediction and receives a corresponding error (Section 3.1). This per-layer of dopamine, and the computation of error and updates can be synchronously parallelized across layers; there are no dependencies across layers. Second, we use forward connectioonn in time to send activations from upper to lower layers (Section 3.2). This provides an information pathway for upper layers communicate with lower layers using activations, rather than error signals, and empirically improves performance. Figure outlines our architecture, unfolded in time.
mirrors the locally homogenous distribution
The AD cell (Figure 3) is where differ most significantly from FF. Its role is to compute the local Q-prediction and TD error. FF is designed to separate real from fake (generated) data, a binary classification task. But the NAc is theorized predict value, regression task, and therefore needs more precision. To achieve this, we introduce an attention-like mechanism for non-linear regression without using error propagation (Section 3.1).
We evaluate AD on 14 discrete and continuous RL tasks from the MinAtar testbed 77). the DeepMind Control Suite (DMC) (64] and classic control environments implemented in Gymnasium [65
"Fooward" and "backward" are widely used in deep learning literature both to describe direction in time and in the order of layers. This can be confusing. For the remainder of this paper, we use "forward/backwar when describing time, and "upper/lower" when describing position among layers, as shown in Figure 2

---

h [3]
121
Ag
h
f
Layers
hn
hg
h2
[1] 1g
[1]
h2
[1]
S Timesteps
2
Figure 2: Network architecture of 3-layer AD network. h! represents the activations of layer at time t, and 8t the input state. The blocks are AD cells, as shown Figure Similar to how dopamine neurons compute and distribute error used by local region, each cell computes its own local TD error used by its updates; errors do not propagate across layers. To relay information, upper layers send activations to lower layers in the next timestep. For example, red shows all active connections at 1.
MinAtar tasks are miniaturized versions of Atari games, and DMC contains continuous control tasks with simulated physics. These environments are complex enough reflect many challenges in modern RL [14), yet remain adequately tractable so not to necessitate extra components like convolutional layers, which may confounding when attributing performance. We benchmark AD against DQN [55, SAC [26] and TD-MPC2 [28) baselines, and conduct ablation studies to examine the effects of the forward connections and additional layers. Our results in Figures 4 and g show that AD learns to solve many of these tasks with comparable performance to the basclines, using just per-layer TD errors. Our code is available at https: / /gi gighub...... com/ sociiicilll aill- 1al--a ai wwo-uoot/aaeee / coott//ddppprrr
To summarize our core contributions:
Are distributed TD error signals sufficient to solve credit assignment and learn complex reward-basee learning tasks? We construct computational example of RL agent that learns using only distributed, per-layer TD errors. This provides evidence that dopamine-distributed signals alone may be enough support reward-based learning in the nucleus accumbens.
We design Q-learning algorithm, ARTIFICIAL DOPAMINE, to train our agent. Like Forward- Forward, AD does not propagate error signals across layers. Unlike FF, we introduce a novel cell architecture to compute Q-value predictions, Q-learning is regression task.
We evaluate our agent on 14 common RL tasks in discrete and continuous control, and show that AD can often achieve comparable performance deep RL algorithms, without backpropagation.
Background
2.1 Reward-Based Learning in the NA
The nucleus accumbens (NAc) plays critical role in the mesolimbic reward system, and is theorized to predict action value [41). The predicted action value is encoded via the firing rate of NAc neurons (23]. which reach dopamine neurons in the ventral tegmental area (VTA) via projections and influence their activity [60). In addition, the VTA receives reward signals from sensory inputs, such as the detection of sugar on the tongue (23) The value predictions and reward signals enable the VTA compute reward prediction errors, i.e. differences between expected and actual reward. These reward prediction errors (more specifically TD errors) are encoded via dopamine, and projected through the mesolimbie pathway regions of the ventral striatum, including the NAc, and lead to synaptic adjustments [60]. When there is positive reward prediction error, the average activity of dopamine neurons increases; when there a negative error, the average activity decreases
*We give a simplified overview of rewerd-based learning in the NAc and its connections to TD learning here to provide intuition: this is not a complete picture of biological reward aoward-based based learning. There exists competing
theories for alternative mappings of the TD learning framework to the brain's 's reward circuit: see Section

---

The exchange of value predictions and error signals between the NAc and VTA sets the stage for TD learning. TD models have shown strong congruency with observed dopamine activity [48], and are widely established as the primary theory for mesolimbic dopaminergic learning.
What is less known is how NAc neurons computationally coordinate learning. iopamine-encoded error signals are distributed (58), which makes credit assign- ment difficult, and there are no other known mechanisms that NAc neurons utilize to communicate er- ror signals for explicit coordination. We illustrate the distribution process in Figure IT Each dopamine neuron projects dopamine along its axon through the mesolimbic pathway, then synaptically releases it in synchronous bursts to the immediate juxtasynaptic area. This causes dopamine concentration in the region to peak, activating dopamine receptors in NAc medium spiny neurons. Dopamine concen- tration levels are locally homogeneous near the synapses [58], but can vary across different regions of the NAc [68] as dopamine neurons need not all fire at once. This supports the distribution of local- ized error signals. While While|Figure ijis clearly not to scale, dopamine neurons are indeed significantly larger than medium spiny neurons. Their large cell body size enables them to support large terminal fields [13, allowing each dopamine neuron to widely distribute its signal to groups of NAc neurons.
the dopamine-encoded error signals to
2.2 Temporal-Difference Learning
Temporal-difference learning (TD learning) methods are family of RI. approaches that aim to solve the problem of estimating the recursively defined Q function [62]. In their most basic form, TD learning methods can be implemented as lookup tables, where an estimate of the Q function is kept for each state-action pair. However, in many practical applications with large or continuous state spaces, a tabular representation of the Q values for all distinct state-action pairs is computationally infeasible. In these cases, function approximation, for example via linear regression neural networks, is necessary to obtain an estimate of the Q function. These methods, called Fitted Q Iteration 4 or Deep Q Learning (44, use the squared temporal difference error as a regression loss [(ss, 0, 8 ) = 5(8, a, 8' )²)², where ò is the TD error, and update a parametric Q function approximation via gradient descent. To prevent the double sampling bias and other instabilities, only Qss, a) is updated and the next states value is estimated with an independent copy Q that is not updated. This is commonly called the bootstrapped Q loss. Given parameters 0, this loss can be written as:
a(s.a,8', 0) (Qo(s,a) - [r(s,a) a) + y max 2o(5",a'))
For more formal description of our TD learning setting, see Appendix A
2.3 Forward-Forward
The Forward-Forwand (FF) algorithm 30 is a greedy multi-layer learning algorithm that replaces the forward and backward passes of backpropagation with two identical forward passes, positive and negative. The positive pass is run on real data; the negative on fake (generated) data. The goal of the model is to learn to separate real from fake data. During training. each layer of the network performs this classification independently using a measure called goodness, which computationally acts as the logit for this binary classification task. Each layer computes its own per-layer error for updating. FF's de facto measure of a layer's goodness g is the sum of its squared hidden activations h4 minus some threshold т, i.e. g = E, h² r. However, due to the simple goodness formula, each layer directly passes its activations to the next, the next layer often trivially learns the identity function. then uses the sum of its last layer's hidden activations as its goodness. This traps the layer in local optimum. To avoid this, FF uses layer normalization [ to normalize the activations before passing them on, which keeps the relative values of the activations, but makes them sum to O.
Compared to BP, a global algorithm that requires the entire network to be updated sequentially, FF local algorithm that updates each layer independently via local, layer-wise losses. Critically, the authors of [30] showed that layers can learn useful representations that help subsequent layers even though they are trained on local errors, without explicit credit assignment. However, for a feedforward architecture, one glaring limitation is that later layers cannot relay any information to earlier ones. Hinton suggests addressing this with a multi-layer recurrent architecture, where for each layer in the network, the input is determined by output of the layer / - 1 and that of l and I 1+ at the previous time step /. 1. This allows for top-to-bottom information flow through the network via activations, which is more biologically plausible.

---

A
[++]]
Concatenate
Copy
Weight layer with tanh activation
tanh
tanh
relu
Dot product
hn
Figure 3: Inner workings of our proposed AD cell (i.e., hidden layer). 11 is the activations of the cell at time t, and @!4º is vector of Q-value predictions given the current state and each action. We compute the cell's activations 1,"" using a ReLU weight layer, then use an attention-like mechanism to compute Q.".. Specifically, obtain Q.¹² by having the cell's 's tanh weight layers, for each action, compute attention weights that are then applied to nn.0) Each cell computes its own error.
3 Artificial Dopamine
ARTIFICIAL DOPAMINE (AD) is a deep Q-learning algorithm that trains deep RL agents using distributed, per-layer TD errors. AD inherits the per-layer predictions and local computation of errors from FF, and adopts the forward-in-time downward connections from FF's recurrent variant. Informally, the intuition is that the per-layer error acts similarly to the locally homogeneous distri- bution of dopamine in the NAc. The per-layer errors can be computed in a parallelized, distributed fashion; each layer computes its own local error, which the neurons of the layer use to adjust weights according their contributions to the error. Similarly, the NAc neurons near the synapses of each dopamine neuron receive the same error signal (encoded via dopamine), which they use to adjust their synaptic weights, according to their previous activity shortly before receiving the emror. During inference, the network uses the average Q-values across the layers to produce final prediction.
Since the NAc theorized to predict action value 23 i.e. Q-value, our prediction task deviates from that of FF, which performs binary classification. In general, predicting action value is nonlinear regression task. To learn this task using neural network, without resorting to the biologically implausible BP, we design an attention-lik, mechanism that learns sets of "attention" weights one set per action head, which introduces nonlinearity. We encapsulate this process in AD cells.
3.1
AD Cell Internals
Our network architecture is composed of layers of AD cells, each of which makes its own Q-value prediction, computes its own local TD error, and updates its own weights. At inference, the final prediction of the network is the average of each cell's Q-value predictions. We
attention-like mechanism [6] that learns a weighted sum of the cell's 's hidden activations to predict the Q-value. The hidden activations passed other cells, but the attention weights and Q-value are withheld. This mechanism simply serves to functionally simulate the complex nonlinear capabilities of biological neurons [00;
not attempting to draw an analog between our mechanism and any biological counterpart, and design choices are primarily made based on empirical performance.
The concept of cell is reminiscent of the design of a recurrent neural network. In our case, there is single cell per layer, so we use the terms somewhat interchangeably for clarity exposition. The vital difference is that no loss information is propagated between cells via BP; that is, there is no BP through time. Instead, the same BP are passed to the cell above (i.e., hidden layer) at the same timestep and to the layer below at the next timestep. In the absence of BP, these activations provide a pathway for upper cells to communicate information to lower cells (see Figure 2). We discuss in more detail how these connections operate in Section
3.2 3.2
Figure presents the cell design. Each cell mechanistically identical and takes two inputs: the hid- den activations of the cell below at the current timestep (or observation, if lowest cell), and the hidden
5

---

activations of the cell above at the previous timestep. It produces two identical outputs, sent to the cell above immediately, and the cell below at the next time step. the start of an episode, the activations from the previous timestep are zeroes. The top cell only receives one input and produces one output.
The attention-like mechanism for Q-value prediction works as follows. Each cell computes its hidden activation, A,"), using the layer above's hidden activations from the previous timestep, + and the layer below's s hidden activations the current timestep, n.4-"1, Specifically, it passes the concatenation of h [++1] and h, [ through ReLU weight layer (shown in Figure to get n,"" The ReLU weight layer multiplies the concatenated input [M!--1] | by its learned weight matrix _ [ W then applies the ReLU nonlinearity function. In parallel, the cell also uses [n to compute "attention" weights, by multiplying it with the learned weight matrix willi then applying the tanh function. Finally, the cell takes the dot product between the output of the tanh layers, tanh(W!! [!] [n!--1, M [ and the hidden activations h!" to compute the cell's Q-value prediction (s,, a). Each cell reuses its internal weights over time; for example the matrix wi!l! is used at each timestep in the first cell. Therefore, the full computation performed by a cell is:
h |++1| 0,
[0] 1 =
! = relu in
[n. 1- )
)
( a) tanh (w!!!
[n]- n " 1.
Optionally, like 30), our architecture supports skip connections between cells. In such cases, the addi- tional input from a skip connection is simply concatenated with [r. [+1]]
before computing 7,"1
3.2 Network Connections
As shown in Figure each cell passes its state to the cell / ++ 1 above at the current timestep 4, and to the cell [ - 1 below in the next timestep + 1. The information flow is strictly unidirectional to match the direction of time flow in RL environments. This is necessary as interacting with the environment happens sequentially, meaning future information will not be available when acting.
Although we do not backpropagate gradients across cells, information does flow from upper layers to lower layers via the temporal connection (forward in time). The upper layers use the connections to communicate with lower layers via activations, which is more biologically plausible 44 Our results in Figure 4 suggest that these connections can greatly increase network performance in the absence of BP. The intuition for adopting these forward-in-time connections is that they are well-suited to take advantage of the temporal structure of the Q-values of trajectories for better learning. Given good policy, the Q-value predictions of a well-trained model should remain stable through each state of trajectory (assuming the dynamics are reasonably deterministic). This means that the Q-value prediction of the current timestep, and the hidden activations used to make this prediction, can often still be useful for predicting the Q-value of the next timestep. In contrast, in FF's experiments on image classification, this effect is forced- FF repeats the same input every timestep, reducing computational efficiency. Our results empirically support the effectiveness of forward connections for Q-learning, particularly in more complex environments.
4 Experiments
The main goal of our experiments is to evaluate whether distributed, per-layer TD errors are sufficient for learning complex RL tasks that are typically only solved with backpropagation and sequential errors. Our criterion of sufficiency is how well the agent learns to solve the given task, measured in terms of average episodic return. Since there are no official standards that define solving these environments, we the performance of established RL algorithms (i.e. DQN, TD-MPC2, and SAC) as the gold standard to compare against. These algorithms are commonly used to solve the environments choose and are known to be strong performers 770 28 26
Like Hinton 30), our aim is to investigate the learning capabilities of a novel algorithm that operates under additional biological constraints, rather than pursue state-of-the-art performance. Thus, we opt

---

for a simple implementation with few extensions. The only extensions we employ are experience replay [45] and the Double-Q learning trick [66] which are standard for deep Q-learning- and skip connections from the input to upper layers for the DMC environments. We do not use convolutional layers, as these more closely resemble the brain's 's visual cortex [40], , and similar structures are not found in the mesolimbic system. We implement our algorithm in Jax 12
Training Process. The training process of our RL agent is based on the standard DQN training algorithm with experience replay [45]. During training. we use a randomly sampled replay buffer that stores every transition observed from the environment. We replay sequences of short length, similar to how other recurrent architectures are trained [29 355, and compute the local updates for each cell sequentially according to the network graph. Since the local updates are distributed and per-layer, they
can be computed in parallel. We provide a formal description of our training process Appendix B
Environments. We run our experiments on 2 RL. benchmarks and 4 classic control tasks, totaling 14 tasks. MinAtar [70] is a simplified implementation of 5 Atari 2600 games: Seaquest, Breakout, Asterix, Freeway, and Space Invaders. The DeepMind Control (DMC) Suite (64] is set of low-level robotics control environments, with continuous state spaces and tasks of varying difficulty. For our experiments, we used discretized action space, following Seyde et al. [61], as our architecture is currently only developed for discrete Q-learning. From the DMC tasks we select Walker Walk, Walker Run, Hopper Hop. Cheetah Run, and Reacher Hard. In addition, we provide results on the classic control tasks Cart Pole, Mountain Car, Lunar Lander, and Acrobot, which we include in Appendix C For more elaborate discussion on these environments and our task choice, see Appendix [
Baselines. On MinAtar, we compare our results against fully-connected DQN to make perfor- mance comparisons more direct and informative. As the original baselines presented in Young and Tian [70] used CNN, we replaced the CNN with fully connected layers and tuned hyperparameters for fairnsss of comparison. We find that the new architecture performs as well as or better than the one presented by Young and Tian [70]. Specifically, we use double DQN with 3 hidden layers of 1024, 512, and 512 units with ReLU activations. This baseline achieves strong performance and has a number of trainable parameters comparable to our networks.
For the continuous control tasks, we show that our method almost reaches the performance of state-of-the-art algorithmic approaches such as SAC (27) and TD-MPC2 (28), which rely on backpropagation. The results were taken from Yarats and Kostrikov [69] and Hansen et al. (28 respectively. We do not change the underlying architectures hyperparameters.
Network architecture and hy hyperparameters. On MinAtar, with forward activation connections. The cell output sizes are 400, 200, and 200. Due to additional connections within cells and from upper to lower cells, this architecture has similar number of trainable parameters as the DQN. On DMC, we use a smaller network with cell output sizes 128, 96. and 96, and discretize the action space following Seyde al. [61]. For more details, see Appendix G For each benchmark, we use the same network and hyperparameters across all tasks to test the robustness of our architecture and learning algorithm.
3-hidden-layer network
Results
We present the results of AD on MinAtar and DMC environments in Figure 4 and compare its per- formance against DQN, SAC, and TD-MPC2. Figure 4 shows the mean episodic return over episodes across 10 random seeds, with standard error. We also provide 95% bootstrap confidence intervals (55) in Appendix D and aggregate statistics in Appendix E We additionally perform ablation studies by removing the forward connections to lower layers, and measuring the performance of a single-layer AD cell. We show both the forward connections and multiple layers contribute to performance (Fig- ures 5 and Finally, we evaluate an implementation of AD that learns distributions over Q-values, based on recent work by Dabney al. (20). and find that AD shows promising results (Figure
Comparison against baselines. We find that AD is able to learn stable policies reliably in all test environments. On MinAtar tasks, our agent achieves comparable performance to DQN on Breakout, and slightly surpasses DQN's performance on Asterix and Freeway, while DQN performs better on
7

---

Freeway
Breakout
Seaquest
Asterix
Space Invaders
DQN
Walker Walk
Walker
Hopper Hop
Cheetah Run
Reacher Hard
000
600
180
TD-NPC2
440
A0
Figure 4: Episodic returns of AD in MinAtar and DMC environments, compared to DQN, TD-MPC2 and SAC. Lines show the mean return over seeds and the shaded area conforms to standard errors. The axes are return and environmental steps.
Freeway
Breakout
Seaquest
Asterix
Space Invaders
AD single layer
AD forward
Figure 5: Ablation study comparing the performance of AD against AD without the forward-in-time connections, and single-layer AD cell. In Seaquest and Asterix, AD achieves qualitatively stronger performance. In Seaquest the line for AD single layer is overlapped by the line for AD no forward.
Seaquest and Space Invaders. On all evaluated DMC tasks, we see that our method's results are close or par with those of SAC and TD-MPC2 when using both backward connections and multiple layers, however, sample efficiency is marginally lower.
We note that the action space discretization we utilize for DMC tasks may complicate the comparison, as it both slows down our training but can benefit algorithms (compare Seyde [6]]). addition, we do not show results on the hardest DMC tasks because of the difficulties in scaling our approach to large action spaces, which leads to rapid growth network parameters.
Overall, our results demonstrate that AD shows biological distributed error signals may allow for coordinated learning in several RL domains. Therefore, further refining and improving the architecture for state-of-the-art RL benchmark performance may be an exciting and promising direction for future work.
Forward connections to lower layers. To further measure the performance impact of the forward connections in time, compared the temporal forward version of the network to 3-layer AD network without the forward-in-tine connections. All other hyperparameters are the same as the 3-layer AD network. As shown in Figure 5 this resulted in moderate drop in performance in most environments, increased variance in the training, and devastating drops in performance on Seaquest and Asterix. These tasks are the most complex out of the five. We provide additional discussion on AD's performance in Seaquest, where the performance difference is most significant, in Appendix(N N These results suggests that the information channels from the upper to lower layers are vital for performance on many tasks. On this same note, AD's ability to achieve similar performance to DQN when the forward connections are added suggest that forward connections may be an effective replacement for backpropagation in certain tasks.
Single-layer performance. Another important question about our proposed architecture is whether the cells learn to coordinate with each other. A concern is that the majority of the learning may be accomplished by the lowest cell, the performance of the multi-layer AD network does not improve over the single cell, it would suggest that we cannot explain AD's performance as result of the cells coordinating their learning using distributed errors. We show in Figure 55 that the multi-layer version of AD outperforms the single-layer at all tasks, and the single-layer cell fails at Seaquest and Asterix.
8

---

Walker Walk
Walker Run
Hopper Hop
Cheetah Run
Reacher Hard
3000
AD
Single 128
Simgle 256
200
200
Single 512
Figure 6: Episodic returns of different-sized single-layer AD, compared to the standard 3-layer AD, Single 128 is single-layer with 128 hidden activations. Overall, increasing the layer size of the single layer does not result in clear increases in performance. Lines show the mean return over seeds and the shaded area conforms to 3 standard errors. The axes are return and environmental steps.
Walker
Walker Run
Hopper Hop
Chestah Run
Reacher Hard
3-0
300
TD-MPC2
SAC
A0-08
Figure 7: Episodic returns of the distributional RL version of AD, implemented with Quantile Regression (QR). Lines show the mean return over 8 seeds and the shaded area conforms to standard errors. The axes are return and environmental steps.
An additional concern along these same lines is whether a wider, single-layer AD cell may achieve the same level of performance as multi-layer AD. In Figure 6 we show that increasing the layer size of a single-layer AD cell does not result in clear increases in performance in DMC tasks. We also experimented with increasing the layer size of a single-layer cell for Seaquest and Asterix from 400 to 600 and 800, and did not find noticeable improvements in either case.
Distributional RL. Recent work by Dabney al. (0) suggest that the brain's value predictors may be learning distributions over future values, rather than just the mean, as previously believed. Dabney al. (20] argue that different dopamine neurons in the VTA may have different scalings for positive and negative reward prediction errors- intuitively, they can be more optimistic or pessimistic- which results in the predictors learning a distribution over the values. Interestingly, this coincides with the development of distributional RL, whose algorithms aim to learn such distributions.
To better align work with the findings of Dabney al. (20]. we additionally implement version of AD that learns distributions over values, and evaluate it on the DMC tasks. Our implementation is based on Quantile Regression (QR) DQNs [19], and requires just simple modification to each AD cell. Rather than predicting single Q-value, each cell predicts 10 Q-values that each match to one quantile of a 10-quantile QR-DQN. The tradeoff is that this requires additional compute.
Our results in Figure suggest that AD may be well-suited for distributional learning. each of the tasks, our agent achieves similar performance to the standard version of AD, and only slightly lags behind on Hopper Hop. This may be a result of the greater sparsity of the Hopper environment, which makes it more difficult for distributional RL algorithms to learn.
Limitations
We proposed an algorithm that trains an RL agent using only distributed TD errors, which provides a computational example of how distributed error signals may be sufficient to solve credit assignment in reward-based learning in the NAc. However, our model does not accurately capture all aspects of the relevant biology. In our model, we use per-layer TD errors as an analogy for dopamine neurons dis- tributing error signals to local regions around their synapses. But unlike in artificial neural networks, which form the basis of our architecture, neurons in the NAc are not clearly organized into layers. AD's hierarchical message passing architecture is design choice we inherit from deep learning prac- tices, and not meant as a mechanistic model of the NAc. Furthermore, activations in biological neurons communicated asynchronously, and can form recurrent loops, which we do not account for.

---

In addition, we make some assumptions regarding biological seward-based learning that are not yet conclusive in neuroscience. Most importantly. we assume that neurons the NAc learn to output action values, and dopamine neurons in the VTA receive reward signals and the predicted action values to compute and return TD errors F While these assumptions are widely supported by research 23 41 46, 47], there exist other theories and empirical results that provide alternative explanations for mesolimbic reward- based learning. Some of these works include Roesch et al. (56), who suggest that on-policy value-based learning (SARSA) better explains dopamine activity than off-policy value-based learning (e.g. Q-learning); Takahashi [33 and Chen and Goldberg [6). who provide evidence that map subregions of the striatum to actor-critic models; Ito and Doya [32 and Weglage et al. [67] who suggest the dorsal striatum signals action values, whereas the ventral striatum signals state values; and Akam and Walton (2 and Coddington et al. [17], who respectively propose how dopamine may be used by the brain to perform model-based and policy-based learning rather than just aalue-based learning. Furthermore, while there is strong evidence that some form of TD learning is used by the brain, mappings between RL frameworks and dopaminergic learning may not be mechanistically accurate even if they are behaviorally accurate.
There are also several technical limitations of our work. First, like other Q-learnnn algorithms, AD requires discrete action spaces. To solve the DMC tasks, which have continuous action spaces, we discretized the action space, following the method used by Seyde et al. [6] (Section 4. While this is consistent with other work in RL, it introduces additional complexities [61] and may not reflect biology. Second, within each AD cell. the number of tanh weight layers grows in proportion to the size of the action space (Figure 3). This limits the scalability of AD for tasks with large action spaces. We can mitigate the effects of this issue using a matrix decomposition trick described in Appendix F. but AD currently cannot scale very large action space DMC tasks like Humanoid or Dog [64]. Third, computational constraints limited the number of runs we perform per task. Additional experiments can further improve the robustness and generalizability of our results.
Finally, our work isolates one system of biological learning, and attempts to provide a computational explanation without accounting for other systems of learning, for example hippocampal contributions to value-based learning [7]. But the brain's 's learning capabilities are likely a result of combining signals from several systems, which may not be divisible [50], and the NAc may be just one part of larger value-based system [8). Unlike backpropagation, the brain utilizes multiple types of learning in conjunction, both supervised and unsupervised, using local and global signals. AD only models one type of learning, i.e. error-driven learning using distributed reward signals to induce local updates. Other biologically plausible algorithms, such ANGC (53) may provide explanations for other forms of learning, which may be critical to building a more complete understanding of biological reward- based learning. Indeed, aspects of Hebbian learning or active inference are likely critical to achieving fully biologically plausible, efficient, and powerful learning system. In that light, we explore learning with just distributed error signals not to rule out the importance of other methods, but to demonstrate that this one principle alone may be sufficient to solve some complex RL tasks nearly as well as BP. We believe that a key to achieving general, human- like intelligence will be the integration of these different learning methods and learning signals; this is an exciting direction we aim to explore in future work.
Acknowledgements
We gratefully acknowledge our sponsors, who support our research with financial and kn-kind contributions: Amazon, Apple, CIFAR through the Canada CIFAR AI Chair program and the Canadian Foundation for Innovation, DARPA through the GARD project, Meta, NSERC through the Discovery Grant and funding reference number Award, the Sloan Foundation, and the Schwartz Reisman Institute for Technology and Society. Resources supporting this research were also provided by the Province of Ontario, the Govermment of Canada through CIFAR, and the sponsors of the Vector Institute. We extend our sincere gratitude to Mete Kemertas for offering insights that significantly accelerated the initial exploration of this work. We also thank Congyu Fang, David Glukhov, Stephan Rabanser, Anvith Thudi, Sierra Wyllie, and other members of the CleverHans and SocialAI labs, as well as our anonymous reviewers for invaluable discussions and feedback.
the Ontario Early Researcher
'Note that AD does not assume action selection need happen in the NAc. Rather, we just take the assumption that the NAc computes the Q-value function, which takes in the state and action as parameters, and performs Q-learning. The action may be provided from another region like the dorsal striatum.
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
[2] Thomas Akam and Mark Walton. What is dopamine doing in model-based reinforcement learning? Current Opinion in Behavioral Sciences, 38:74-82 82, Apr 2021. doi: 10.1016/f.cohea.. 10.00.
[3] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances neural information processing systems, 32, 2019.
[4] András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and single sample path. Machine Learning, 71 71:89-129, 2008
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 140x.v/409.0444 0,73, 2014.
[7] Akram Bakkour, Daniela J Palombo, Ariel Zylberberg, Yul HR Kang, Allison Reid, Mieke Verfaellie, Michael N Shadlen, and Daphna Shohamy. The hippocampus supports deliberation during value-based decisions. elife, 8:e46080, 2019.
[8] Ulrike Basten, Guido Biele, Hauke R Heekeren, and Christian J Fiebach. How the brain integrates costs and benefits during decision making. Proceedings of the National Academy of Sciences, 107(50):
2010.
[9] Hannah M. Bayer and Paul W. Glimcher. Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal. Neuron, 47(1):129-141, July 2005. ISSN 08966273, doi: 10. 1016/j.neuron..220 05.020. URL httpps://linkinghubb. elsevier com/retrieve/piil $0896627305004679
[10] David Beniaguev, Idan Segev, and Michael London. Single cortical neurons as deep artificial neural networks. Neuron, 109(17):277-2733..e) September 2021. ISSN 08966273. doi: 10. neur/onneuron.222 2021 .07.002. URL https://1inkkigguub. elsevier com/retrieve/pp... 50896627321005013
[11] Matthew Botvinick. Jane Wang, Will Dabney, Kevin J. Miller, and Zeb Reinforcement Learning and Its Neuroscientific Implications. Neuron, 107(4):603-66) 616, August www.: URL ltttps///lnkkinghu)) elsevier com http://rrroorrr///////6662200000
Deep
2020. ISSN 08966273. doi:
[12] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://github: wwt. wowme//oo////a..
[13] S-R Cajal. Les nouvelles idées sur la structure du système nerveux chez | l'homme et chez les vertébrés. C. Reinwald & Cie, libraires-Editeurs 1894.
[14] Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pages 1373-1383, PMLR, 2021.
[15] Chun-Chuan Chen, RN Henson, Klaas E Stephan, James M Kilner, and Karl Friston. Forward and backward connections in the brain: dem study of functional asymmetries. Neuroimage, 45(2):453 462, 2009.
[16] Ruidong Chen and Jesse Goldberg. Actor-critic reinforcement learning in the songbird. Current Opinion in Neurobiology, 65:1-9, Dec 2020. doi: 10. hww.://.comb.020.0000.

---

[17] Luke T. Coddington, Sarah E. Lindo, and Joshua T. Dudman. Mesolimbic dopamine adapts the rate of learning from action. Nature, 614(7947):294-000 Jan 2023. doi: 10. 10.1038/44558-022---00
[18] Anne G E Collins and Michael J Frank. Surprise! Dopamine signals mix action, value and error. Nature Neuroscience, 19(1);3-5, January 2016. ISSN 1097-6256, 1546-1726. doi:
n0.1038/nm.4207. URL http::tt///// / www. www.nnttuee... wwature.com/aaeee arti ceaeiicceeennnneee
[19] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[20] Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick. A distributional code for value dopamine- -aase reinforcement learning. Nature, 577(7792):671-637 2020.
[21] Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit assignment problem without a backward pass. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4937-4955. PMLR, 17-23 Jul 2022. URL https html
7/ p/poccediiggs.o press/pes///vse wwtp////elafeeerre222222
[22] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141112,, 2012.
[23] Paul W Glimcher. Understanding dopamine and reinforcement learning: the dopamine re- ward prediction error hypothesis. Proceedings of the National Academy of Sciences, 108 2011.
[24] Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-63, 1987.
[25] Jordan Guerguiev, Timothy Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. eLife, 6:e22901, December 2017. ISSN 2050-084X. doi: 10.7554/eLife. 22901. URL httt:pp:/eelliieeeiiencsss.srr wttp://rrrraaaatteeeeeeeaaa/////
[26] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. 2018.
[28] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control, 2023.
[29] Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
[30] Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint 23X1v:2212.13345, 2022.
[31] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022. TtLLhtt::// //jmlr. org/ pttppp/aaa//////// 1342. html
[32] Makoto Ito and Kenji Doya. Distinct neural representation the dorsolateral, dorsomedial, and ventral parts of the striatum during fixed-and free-choice tasks. Journal of Neuroscience, 35()): 3514, 2015.
[33] Dmitry Ivanov, Aleksandr Chezhegov, Mikhail Kiselev, Andrey Grunin, and Denis Larionov. Neuromorphic artificial intelligence systems. Frontiers in Neuroscience, 16:1513, 2022.
12

---

[34] Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without feedback. arXiv preprint arXiv:2209.11883 2022.
[35] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[36] Daniel Kunin, Aran Nayebi, Javier Sagastuy- -Beena, Surya Ganguli, Jonathan Bloom, and Daniel Yamins. Two routes to scalable credit assignment without weight symmetry. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5511-5521. PMLR, 13-18 Jul 2020. URL URL http:/tt:///prrcedddnng
mlr wwwpppesssvvveee www1///uiiinii html
[37] Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propa- gation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15, pages 498-515. Springer, 2015.
[38] Timothy P Lillierap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature communications, 731):1 2016.
[39] Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335 -3633,666,,,0 June 2020. ISSN 1471-003X, 1471-0048. doi: 10. 10000//4183-00000027774 URL https: http//wwwwwttutt nature. com/
[40] Grace W Lindsay. Convolutional neural networks model of the visual system: Past, present, and future. Journal of cognitive neuroscience, 33(10):2017-2031, 2021.
[41] Francesco Mannella, Kevin Gurney, and Gianluca Baldassarre. The nucleus accumbens as nexus between values and goals in goal-directed behavior: review and hypothesis. Frontiers in behavioral neuroscience, 7:135. 2013.
[42] David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[43] Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training self- meff-modiiying neural networks with differentiable neuromodulated plasticity, 2020.
[44] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[45] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[46] Genela Morris, Alon Nevet, David Arkadir, Eilon Vaadia, and Hagai Bergman. Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 10): Jul 2006. doi: 10. 10310/8/n 1743.
[47] Yael Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3): 139-154, Jun 2009. doi: 10. 2006//..mp.20000 12.005.
[48] John P O''Doherty, Peter Dayan, Karl Friston, Hugo Critchley, and Raymond Dolan. Temporal difference models and reward-related learning the human brain. Neuron, 38(2)332--337, 2003.
[49] Randall C O'Reilly. Biologically plausible error-driven learning using local activation differ- ences: The generalized recirculation algorithm. Neural computation, 8(5):895 938, 1996.
[50] Randall O'Reilly, Yuko Munakata, Michael Frank, Thomas Hazy, et al. Computational cognitive neuroscience, volume 1124. PediaPress Mainz, 2012.
13

---

[51] Alexander Ororbia and Daniel Kifer. The neural coding framework for learning generative models. Nature communications, 13(1):2064, 2022.
[52] Alexander G. Ororbia. Brain-inspired machine intelligence: A survey neurobiologically- plausible credit assignment, 2023.
[53] Alexander G Ororbia and Ankur Mali. Backprop-free reinforcement learning with active neural generative coding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 29-37, 2022.
[54] Thomas Parr, Giovanni Pezzulo, and Karl J Friston. Active inference: the free energy principle in mind, brain, and behavior. MIT Press, 2022.
[55] Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. arXiv preprint arXiv:2304.013//1 2023.
[56] Matthew R Roesch, Donna Calu, and Geoffrey Schoenbaum. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10(12):1615-122, Nov 2007. doi: 10.1038/nn2013.
[57] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by pack-propag errors. Nature, 323(6088):533-532) 1986.
[58] Wolfram Schultz. Predictive reward signal of dopamine neurons. Journal of neurophysiology. 1998.
[59] Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 25935396, 1997.
[60] Susan R Sesack and Anthony A Grace. Cortico-basal ganglia reward network: microcircuitry. Neurepsychopharnaccllyyy 35(1):27-47, 2010.
[61] Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 2021.
[62] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9-44, 1988.
[63] Yuji Takahashi. Silencing the critics: Understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in Neuroscience, 2(1):86-9)) 99,6999,12 Jul 2008. doi: 10.3389/nurr.00.0014.2000
[64] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, Abs//880.0006)) 2018. URL http://arxii org/ abs/1801. 00690
[65] Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré. Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL /ttttt:tp//tentiieeeeeeeett http:///eeeooooooorrrrrrr/////// 8122702
[66] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q q-leamiig. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[67] Moritz Weglage, Emil Wärnberg, Iakovos Lazaridis, Daniela Calvigioni, Ourania Tzortzi, and Konstantinos Meletis. Complete representation of action space and value in all dorsal striatal pathways. Cell reports, 36(4). 2021.
[68] Mark Wightman, Michael LAV Heien, Kate M Wassum, Leslie A Sombers, Brandon Aragona, Amina S Khan, Jennifer I. Ariansen, Joseph F Cheer, Paul EM Phillips, and Regina M Carelli. Dopamine release is heterogeneous within microenvironments of the rat nucleus accumbens. European Journal of Neuroscience, 2677):0046-254, 2007.
14

---

[69] Denis Yarats and llya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https //github.com catp///aaiaaaarrra///ptrrraaaaaa 2020.
[70] Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible 2019.
reinforcement learning experiments. arXiv preprini
[71] Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, et al. Toward next- generatonn artificial intelligence: Catalyzing the neuroai revolution. arXiv preprint arXiv:2210.08344 2022.
15

---

A Formal Definition of TD Learning
We consider standard discounted infinite horizon Markov Decision Process (MDP) setting with states S, actions A. a transition kernel ss a), reward function r S x A + R and a discount factor 7 [0, 1]. This is a typical setting for reinforcement learning 62
The goal of TD learning is to obtain policy x(a|s) that maximizes the discounted future sum of rewards. The value function Q S x A - IR measures how valuable given action E A is in given state 8 E S, and can be used to directly compute policy. It defined via a recursive r(s,a)
Eg p(8'|s, a) En (aa's' Qss)) aa). When experience in the form of 8, a, r; s tuples becomes available from interaction with the MDP, the Q function estimates are updated using the TD error 0(s, a, s') r(s. a) + y Ea π (a'ls' ( a') Q(s, The update rule is Qu+1($,0) a) + Qu a) + and(s, a, s'), where specifies the learning rate.
formula Q(s,a)
B Training Process
Algorithm AD Q-Learning
Initialize replay memory ID to capacity N Initialize action-value functiion Q!I,..., Q141 with random weights , 0.4] Initialize target function for episode .M do
Q(r) with weights g(3 g(t) for every E {l,..., L}
Initialize $1 = iniiiill state
for f= 1,7 do
With probability select random action
otherwise select a, arg max, I E Q (st, a; 0l1)
Execute action 0f in emulator and observe reward r++1 and next state 81+1 Store experience (st, (t, T++1, 8e+1) in DD
Sample minibatch of K-sized episodic sub-trajectories
(3j- K; 0j- K Tj K+, 83- -)) (8), (3411194)))))44
from with uniform randomness
Replay states Sj-K...., 8--1 through online network obtain layer activations
jj-1...., h 1
Set current layer input x + for layer / I,....... do
ry+1 1 if episode terminates at step
Set TD target y,"
Tj+1 max. Q19 (#j+1, (il (xi+1, a; 019) oW') otherwise
Set prediction yj Q!1l (st,, ay; (l4) using and last activations h ((-1) and n
Perform gradient descent step on (Y y5)² with respect to the layer parameters 0/4)
Set input for next layer +- h// (I) end for
Every C' steps reset g0'l (( for every E {1...., L} end for end for
C Experiments on Classic Control Environments
Beyond MinAtar and DMC, we also evaluate AD on classic control environments: Cart Pole (also known inverted pendulum), Mountain Car, Lunar Lander, and Acrobot, and compare its performance against DQN. We show our results in Figure X
The main purpose of these experiments is to further support AD's robustness across different RL environments, and indirectly compare its performance against Active Neural Generative Coding (ANGC) [53]. another biologically-insphe deep RL algorithm that provides more plausible
16

---

Acrobot
Cartpole
Mountain Car
Lunar Lander
-350
300
-10
110
200
-200
300
-140
100
-250
CeantiL's
420
AMSC's
100
-150
-100
-500
AD
-20
3.0
26
2.0
1.0
0.4
Figure 8: Episodic returns of AD, CleanRL's DQN, and ANGC's DQN. Lines show the mean episode return over 10 seeds and the shaded area conforms standard errors of the mean.
alternative to backpropagation. Ororbia and Mali [53 evaluates ANGC in environments: Cart Pole, Mountain Car, Lunar Lander, and custom robot-arm-reaching environment.
In addition, we implement DQNs with different hyperparameters baselines for comparison: first is CleanRL's reference DQN for classic control problems [I1, and second is the DQN tuned by 53 for comparison against their ANGC agent, which we refer to ANGC's DQN. The CleanRL DQN provides a well-established, publicly-vetted baseline, whereas ANGC's DQN serves as a reference point for us to indirectly compare AD's performance with ANGC.
To better demonstrate AD's robustness across environments, we use the same set of hyperparameters for each of the environments. Specifically, and 96, fixed learning rate of 2.5e with Adam optimization, an exploration fraction of 0.2, and final epsilon of 0.05. All other hyperparameters are the same as our network for MinAtar, shown in Appendix G As neither CleanRL nor ANGC's DQN uses a learning rate scheduler, we also removed ours for better comparison. CleanRL's DQN is a 2-layer network with 120 and 84 units; the same DQN is used for all 4 environments. ANGC uses different 2-layer DQNs for each of their environments, which they tuned for environment speciffc performance. Their Cart Pole DQN has 256, 256 hidden units; Mountain Car has 256. 192, and Lunar Lander 5 512, 384. For Acrobot, we used the same hyperparameters as the Cart Pole DQN, given the similarity of the environments, which worked well. For more hyperparameter details, we refer the reader to 31 and [53] both works present excellent detail.
2-layer AD network, with output sizes 128
We made two adjustments when reproducing CleanRL and ANGC's DQNs. First, since AD utilizes double Q-learning improve its learning stability, for better comparison, we also enhanced both DQNs with double Q-learning, given that it is a simple enhancement without additional computational costs. Second, specific ANGC's Cart Pole DQN, we found that the agent's learning has very high variance with the given hyperparameters, and the agent does not consistently achieve high performance. To counter this, we decreased the learning rate from the provided 5e to 2.5e and increased the target network update frequency (referred to C in (33) from 128 to 500, which is CleanRL's value. This improved both the agent's stability and average return.
Our results in Figure g show that AD achieves strong performance across all 4 tasks, learning more stably and achieving higher average episodic return than the DQNs in all environments. Most notably, in comparison to DQN, AD more consistently reaches the maximum 500 return in Cart Pole, and achieves significantly higher return in Mountain Car. It also surpasses DQN's performance slightly in both Lunar Lander and Acrobot. These results suggest that AD is highly competitive against ANGC in these tasks. However, we note that as ANGC and AD learn with different signals, they are not competing approaches, and may possibly be integrated to achieve greater performance- we also look forward to exploring this potential future work.
D Bootstrap Confidence Intervals
We show the results of our main experiments with 95% bootstrap intervals in Figure 9 following the recommendations by 55 to encourage better comparison of RL algorithms and reproduction of their results. For the MinAtar environments, we also show the ablation study results with 95% bootstrap intervals.
17

---

Freeway
Breakout
See quest
Asterix
Spaceinwaders
DON
Cereeesdeettte Sostomii Hopper
Walkerr
Walk
Cheetah Run
AD
200
Tb MP2
SAC
Figure 9: Episode return across the Minatar and Mujoco environments. Shaded area shows a 95% bootstrapped confidence interval around the mean computed over 20 seeds for AD and 3 seeds for TD-MPC2 and SAC respectively.
DQN AD single layer
0.6
0.8
1.0
0.6
0,8
1:0
0.6
0.3
DON Normal ized Score IQM
Median
Mean
TD-MPC2
AD
1.12
1.20
1.04
1.08
1.04
1.20
SAC Normalized Score
Figure 10: RLiable aggregate statistics across the Minatar and Mujoco tasks.
E
Aggregate Statistics
Using the RLiable library provided by Agarwal al. [. we aggregate the performance of AD and baselines on the Minatar and Mujoco tasks. Since the MinAtar and DMC environments do not offer human-normalized scores, we used the performance common baseline algorithms (SAC and DQN, respectively) for normalization.
We find that AD is within the confidence interval of TD-MPC2 on Mujoco and DQN on Minatar (compare Figure IO). It shows better mean and median performance than SAC on the Mujoco tasks. IQM excludes the harder hopper environment, which results in slightly worse performance of AD on this metric, as our method shines in this task.
F Improving Computational Efficiency in AD Cells
limitation of AD cells that the !!lll matrix scales in the size of the action space |A and the size of the hidden layer, When training on more complex environments with large action spaces and require larger hidden layers, Will can become expensive compute and learn. One trick employed in DMC environments to improve the computational efficiency our AD cells is to learn product of two smaller matrices, one |A| x & and the other l: x d where & is
the wi matrix small constant. For small k the number of parameters the matrix product is significantly smaller
than the original will] all matrix, especially when either |A| or d are large, allowing for more efficient learning at the cost expressiveness. We found that empirically the DMC environments using k 8 does not notably negatively impact performance while reducing run times.
18

---

G Model Hyperparameters for MinAtar
We show the hyperparameters of the AD network and DQN used in our MinAtar experiments.
Hyperparameter
AD
DQN
Learning rate
10
10
Exploration fraction
0.1
0.1
Final epsilon at end of exploration
0.01
0.01
Loss function
Mean squared error
Mean squared emror
Max gradient norm
1.0
1.0
Batch size
512
512
Gamma
0.99
0.99
Training frequency
4
4
Replay buffer size Target network update frequency
x 10
x 106
1000
1000
We also use a learning rate scheduler for both networks, which linearly increase learning rate to 10-4 4 in 500000 steps, then cosine decay until 3 x 10 over remaining training steps. These hyperparameters were jointly tuned and shared by both the baseline DQN and AD network.
H Model Hyperparameters for the DeepMind Control Suite
We show the hyperparameters of the AD network in our DMC experiments.
Hyperparameter
AD
Learning rate
2.5 10-4
Exploration fraction
0.25
Final epsilon at end of exploration
0.01
Loss function
Mean squared error
Max gradient norm
1.0
Batch size
512
Gamma
0.99
Training frequency
Replay buffer size
x 106
Target network update frequency
1000
Environment Selection
We run our experiments on standard RL benchmarks and 4 classic control tasks.
MinAtar [70] is an simplified implementation of 5 Atari 2600 games: Seaquest, Breakout, Asterix, Freeway, and Space Invaders. We use version for all environments. The input frame size is 10 x 10, and the n different objects in each game are placed on separate frames, resulting in a 10 x 10 n input observation. All tasks have discrete action spaces, with up to 6 actions.
The DeepMind Control (DMC) Suite (64 is set of low-level roboties control environments, with continuous state spaces and tasks of varying difficulty. For our experiments, we used discretized action space, following Seyde et al. [61]. as the architecture is currently only developed for discrete learning. From the DMC tasks selected Walker Walk, Walker Run, Hopper Hop, Cheetah Run, and Reacher Hard. These tasks were chosen based on the size of the action space, as our discretized action heads currently scale exponentially with increasing action dimension. Using better fine-grainer motor control is a valuable path for future work; for now we present the simplified variant as a proof of concept that algorithm is able to handle wide range of tasks and input modalities.
chose these environments because they reflect challenges solving complex, non-linear control tasks, from which meaningful algorithmic insight can be obtained [14] but are not so complex they require additional components like convolutional layers to solve. This makes them excellent testbeds
for novel learning approaches. Although the DQN baselines published by Young and Tian [70] are
19

---

Dataset
Hidden activations
Layer 1
Layer
Layer 3
Layer 4
MNIST
500
0.9848
0.9862
0.9865
0.9867
MNIST
1000
0.9834
0.9860
0.9866
0.9870
MNIST
2000
0.9834
0.9862
0.9878
0.9874
CIFAR-10 10
500
0.4715
0.5234
0.5394
0.5501
CIFAR-10 10
1000
0.4750
0.5358
0.5496
0.5588
CIFAR-I0
2000
0.4841
0.5440
0.5610
0.5725
Table 1: Test accuracy of each layer of 4-layer AD networks trained MNIST and CIFAR-10. Each layer has the same number of hidden activations; for example, the first row refers to an AD network with layers of 500, 500, 500, and 500 activations. Since each layer makes its prediction, can easily see how performance increases per layer.
obtained using convolutional neural network (CNN), larger DQN with only fully-connected layers can solve all tasks with comparable or stronger performance.
In addition, we provide results the classic control tasks Cart Pole, Mountain Car, Lunar Lander, and Acrobot, which we include in Appendix
Experiments on MNIST and CIFAR-10
To further investigate the generalizability of AD across different learning tasks, we also evaluate AD on two supervised learning datasets, MNIST and CIFAR-10, 10, and present our results here. Q-learning a regression problem; adapt our architecture solve these classificatinn problems, we simply change the loss function to cross entropy. and make each of the tanh weight layers correspond to a class, instead of an action. Using a 4-layer AD network with output sizes 2000, 2000, 2000 and 2000, we achieved 98. 74% test accuracy MNIST, and 57. 37.25% test accuracy on CIFAR-10. These results are line with the results achieved by Forward Forward (30]
We show our results with different sized networks in Table I As each layer in the network makes its own prediction, we additionally show the test accuracy of every layer. Remarkably, for almost every network size, the test accuracy steadily increases per layer. Like our experiments in RL, this supports that AD cells can learn coordinate with each other for better performance without the need to backpropagate етrоr signals across layers.
For clarity, want to note that these experiments performed simply to add another perspective to evaluate the robustness of AD's 's learning across tasks. AD is intentionally designed for Q-learning, rather than supervised learning, it imitates biological processes of reward-base learning.
K Compute Resources
We ran our experiments on shared scientific computing cluster using assortment of CPU and GPUs. Each instance is run with a GPU (to improve the speed training the neural network) and 12 CPU cores (to improve the speed of the environment simulation), and 20GB of RAM. On an Nvidia RTX 2080 GPU, a full training run of AD takes approximately 5 hours on the MinAtar environments, and 3 hours on the DMC environments. On an Nvidia A100 GPU, the takes approximately 3.5 hours on MinAtar, and 2.5 hours on DMC. We additionally expended compute resources for hyperparameter tuning and ablation studies. In total, our experiments consumed approximately 9000 hours of compute using the aforementioned instances.
While we did not prioritize computational efficiency in the current design of our algorithm, we note that like Chen et al. [15], Lillicrap et al. (38] Journé et 34 and other pooloically-pllusibbe deep learning algorithms that do not suffer from the weight transport and update locking problems, our algorithm can be adapted for neuromorphie hardware that keep data local to computational units 33 These hardware have the potential for significant energy efficiency gains.
20

---

L Usage of Existing Assets
We utilize several existing assets in both the implementation of our algorithm and our experiments.
In Table 2 we provide their versioning, licensing and and URL for easier reproduction of our results.
Asset
Version
License
URL
Jax [12]
0.4.11
Apache 2.0
https//ggthhb.ccom/ooogl///////
MinAtar [70]
GPL-3.0
https//gghhbbbcoom//ccyoogggggnn
DMC [64]
1.0.16
Apache 2.0
http://ghhhbbboommoooggeeee//p// control/
Gymnasium 65
0.28.1
MIT
Table 2: Existing assets used.
M
Social Impacts
Our work introduces a new learning algorithm that computationally demonstrates how the credit assignment problem may be solved in the nucleus accumbens, where reward prediction errors are synchronously distributed. This research direction may yield significant positive impacts to healthcare and the social sciences. By improving our understanding of biological reward-based learning, our research has the potential to inform novel therapeutic strategies for neurological disorders characterized by dysfunctional reward processing mechanisms. By better aligning AI with biological intelligence, we also provide social science with more faithful AI agents for simulating and studying human social behavior, a practice that is increasingly adopted.
However, aligning neuroscience and AI is not without risk of negative social impacts. The devel- opment of more faithful AI agents may encourage social sciences to increasingly replace human subjects, as Al agents may be cheaper or more malleable to certain experimental setups, and exoge- nous variables may be more easily controlled for. Yet Al agents inherently biased by the data they are trained upon; replacement of human subjects may further marginalize underrepresented communities, which is already large challenge in AI fairness research.
In addition, deep reinforcement learning research, such as our work, consumes considerable amounts of computational power and energy (see Appendix K). While such research often yields valuable insights into intelligence, these gains should always be balanced against the carbon emissions caused by large-scale experiments.
N Discussion on Seaquest
In Seaquest, where the performance difference between 3-layer AD, AD without forward-in-time connections, and single layer AD is most apparent, the latter agents' struggle may be attributed to the failure of learning the resurfacing mechanism, which traps it at local minima. In this environment, the agent must regularly resurface while carrying a diver to replenish oxygen. In the short term, the agent can acquire more reward if prioritized attacking enemies, rather than managing oxygen, but if it runs out of oxygen the episode ends. We observed that the AD agent without forward connections and the single layer agent both struggled to manage oxygen, often resurfacing randomly or without first acquiring diver. In contrast, the AD agent with the forward connections learns to manage their oxygen more optimally, and often sacrifices short-term rewards to maintain higher oxygen. In fact, during some trials the AD agent became ultraconservative with oxygen, which led to the agent surviving on average over 1500 timesteps per episode over several thousand episodes before it changed strategy. Interestingly, this same behavior was not observed in DQN agents, whose average episode lengths are consistently under 800. This suggests that AD has the potential to learn completely different policies from DQNs.
21

---

Additional Related Work
Biologically inspired learning algorithms seek to reconnect machine learning with neurobiology. This supports better transdisciplinary collaboration across ML and neuroscience, which may be essential advancements in both biological and artificial intelligence research 77].
Similar to our work, Ororbia and Mali [53] proposes deep Q-learning algorithm, ANGC, based on the Neural Generative Coding framework [51], which is centered around the idea of active inference 54 We indirectly compare the performance of AD and ANGC in Appendix C Also closely relevant, Guerguiev al. 25 proposes BP alternative that solves credit assignment for supervised learning tasks, and show promising results on the MNIST dataset (22). They point out that current neuroscience lacks sufficient understanding of how credit assignment performed in the brain, and provide method to train neural networks using segregated dendrites.
Many other related works focus other aspects of biological plausibility in deep learning; mention a few here for the interested reader. Lillicrap et al. (39], Lillicrap et al. [38]. Kunin et al. [36,, Akrout et al. [3] and Lee et al. (37] proposed algorithms that tackle the weight transport problem and non-locality of errors, which have long plagued backpropagation [24]. Journé et al. (34] adopts
Hebbian learning approach address weight transport, locality and update locking. Miconi al. (43
improves the plasticity of neural networks, inspired by the brain's 's mechanisms of neuromodulation.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our paper aims to provide a potential explanation to how distributed error signals in the mesolimbic dopaminergic system can support credit assignment in biological construct learning algorithm that trains neural network solve reinforcement learning tasks using distributed enror signals, without sequential propagation of error between layers of the network. Our experimental results support claim by demonstrating that algorithm solve range of environments with comparable performance to backpropagation, and our ablation studies provide further insights how each part of our algorithm contributes to these results.
reward-based learning.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide an explicit Limitations section (Section 6) to discuss the limitations of our work.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work does include theoretical results.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide formal descriptions of our architecture (Section 3 and learning algorithm (Section B). as well as hyperparameters for each environment (Sections G and to aid reproduction. We also provide our code, and all environments we evaluate our algorithm upon are open source.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We publicly release code via a GitHub repository (see Section I and all enviromments we evaluate our algorithm upon are open source.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details regarding environments, network architectures and hyperparameters Section and Appendices G and
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
23

---

Answer: [Yes]
Justification: We provide error bars that show 3 standard errors of the mean for all experiment results (Section[5). We also provide 95% boostrap confidence intervals (Appendix ( The error bars are derived over 10 trials for main experiments, and at least 3 trials for some ablation studies that require higher compute.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide a description of the compute instances we utilized, as well as number of hours required in Appendix K
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://neurip::: httprrippppcppppiiipcceeee httpp///cce.hhhhcccccsleeeeaaaa.
Answer: [Yes]
Justification: Our work fully conforms with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include a Social Impact section (Appendix M) discuss both positive and negative potential impacts of our work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do release any data models that have high risk of misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We include Appendix to the existing assets we used, their versioning, license, and URL. We also provide cilations to the assets we used.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only new asset we introduce is our codebase, which we publicly release with usage instructions.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments or research with human subjects.
24

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve human subjects.
25


Reviews:
[{"review_id": "bd7ZGqLsFk", "reviewer": "Reviewer_K1xv", "summary": "An novel, biologically plausible RL algorithm based on DQN and Fast-Forward called Artificial Dopamine is introduce. The algorithm is inspired by Dopamine pathways in the human brain and uses synchronous, locally computed per-layer TD errors for training. Additionally, a novel recurrent network architecture is proposed where layer outputs are used as inputs to the respective preceding layer at the next time step. The approach is evaluated on a range of discrete and continuous RL tasks.", "questions": "- L252: \"Since the local updates are distributed and per-layer, they can be computed in parallel.\" The temporal dependency between time-steps due to the recurrence is still present. So, does this actually translate to improved runtimes?", "limitations": "The Limitations such as the use of the use of a layered structure, and limited proof for the method implemented (Q-learning) occurring in the brain are discussed in detail.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Investigating biologically plausible learning is an interesting field of research and surely significant. The manuscript is well written and reads very smoothly. Explanations of biological phenomena are clear and informative, and foremost rigorously cited. The authors clearly distinguish which parts are inspired by biology and which are deliberate choices due to empirical reasons.  Limitations and societal impact are discussed thoroughly.", "weaknesses": "Minor Issues:\n- L 116: \"and is\" -> \"and are\"\n- L134: \"in many practical applications, such as TD learning in the brain\". Makes it sound like this a proven fact that there is TD learning taking place in the brain...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback, and we’re glad to hear that you appreciate the significance, scientific rigor, and clarity of our work.\n\nWe have fixed the typo on L116, and reworded L134 as “…in many practical applications with large or continuous state spaces…” to avoid confusion; thank you for pointing out these issues.\n\nThe reviewer also raised an insightful question: whether the temporal dependency between time-steps due to recurrence would prevent parallelization (and thus improved runtimes). For most recurrent architectures trained with backpropagation-through-time, this is indeed the case; the updates of each layer at each time step must be sequentially computed. However, in AD, unlike backpropagation-through-time, we do not propagate any error signals back along the recurrent connections, thus there does not exist any such dependencies during learning. To elaborate, the forward-in-time recurrent connections only serve to pass activations from upper layers to lower layers; no error information is passed back through these connections. Therefore, the recurrent connections do not prevent parallelization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9DHoGqDFMH", "reviewer": "Reviewer_Sdew", "summary": "In this work, the Authors aimed to propose a computational model that is consistent with classical functions assigned to select regions in the reward system in the brain. To this end, they build upon George Hinton’s Forward-Forward algorithm, extending it with the ability to robustly generate continuous (instead of categorical) values and endowing it with the DQN loss. The Authors show that the proposed algorithm reaches competitive scores on a panel of standard RL benchmarks, and discuss its potential mapping on the brain circuitry.", "questions": "N/A", "limitations": "Related literature on neuroanatomy and computational models of the reward system are not considered.\n\nThe proposed model is not contrasted against other candidate models and thus may be uninformative.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The text is clear and almost always straightforward. Many key statements are accompanied by reasonable disclaimers; as a result, the claims are seldom overstated. The paper has a reasonable description of the related (used) algorithms. The figures are high-quality, which makes the paper stand out among many other submissions.\n\nThe experiments are performed on a sizeable panel of benchmarks. Wherever the baseline design had to be altered to ensure a fair comparison with the proposed model, correct adjustments are performed including the (hyper)parameter search for the altered models.\n\nThere may be a subcommunity interested in these results. After George Hinton’s successful presentation of his Forward-Forward algorithm at NeurIPS 2022, many people present at this keynote lecture got interested in the topic.", "weaknesses": "The paper’s neuro premise (Sections 1; 2.1) reads as if the roles (or even activities) of the brain regions involved in the brain circuit are known undebatable. Even though this impression is then refuted in Section 6 (which is commendable), most of the paper operates under this assumption. While the paper correctly cites some classical literature on the topic, the debates over the brain region roles proposed in these works led to more nuanced models, e.g. the mappings of the actor-critic model on the reward circuit, and then to an overall notion that such mappings represent a bird-eye view of the problem and may not necessarily be mechanistically accurate. It would be nice to explore and discuss this uncertainty, as it may alter the ground truth for this work.\n\nThe paper’s comp neuro premise limits itself to the Forward-Forward algorithm. Although some of the other related work is mentioned in Section 7, these and other works have not made it into the model’s design. Meanwhile, the field of biologically plausible learning, including RL, is immense. There’s a host of work on predictive coding and local learning rules; the new works are submitted to (almost) every major ML conference. There are noteworthy works on the computational role of dopamine and RL in the brain, including the works on \"Backpropamine\" and DeepMind’s modeling of distributional dopamine coding. In that sense, it’s unclear why the current submission focuses on a single model, mostly validated within the ML community while ignoring the host of models validated within the computational neuroscience community. This may be mitigated by the Authors’ comment that their goal was to present a model consistent with neuroanatomy, which is valid, but brings me to the next point.\n\nA single model consistent with data doesn’t preclude the existence of multiple other models, potentially with different mechanics, leaving the chance that the proposed model has little to do with the biological reality. Indeed, a Monte Carlo – or even brute force model – will eventually learn an approximately correct Q-function and won’t need backprop / coordinated feedback signal to do so. While – and I would like to stress that – the results in this paper are correct and the wording is correct as well, arguably, to learn something from this result, one would need to perform multiple additional comparisons between the (versatile, detailed) neuroanatomical structure of the reward circuit and a panel of prominent computational models. As such comparison is not performed within the scope of this work, it is arguably too early to recommend this paper for publication in NeurIPS at this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We noticed that this review is nearly identical to an earlier review we received at a different conference. Building on the previous feedback, we have significantly updated our manuscript with additional experiments, ablation studies, and expanded discussions, including new experiments and algorithmic updates aligning with DeepMind’s distributional dopamine coding (Section 5). We are thus surprised to receive the same review with a decreased score. Assuming you are the same reviewer, we wonder if there might have been a transpositional error or if the old version of our manuscript was reviewed by mistake?\n\nIn either case, we’d like to directly respond to your concerns, and highlight the relevant updates we made since our last submission that aims to address these concerns.\n\n**The neuro premise in Sections 1 and 2.1 reads as if the roles of the brain regions involved in the brain circuit are known undebatable. This is refuted in Section 6, but the discussion is inadequate.**\n\nWe acknowledge this was a weakness in our previous submission. We expanded our Limitations section to explicitly state the assumptions we make regarding the biological mapping and included additional discussions on alternative mappings of RL frameworks to the reward circuit. We additionally discussed works by Takahashi et al. (2008), Chen & Goldberg (2020), Morris et al. (2006), Niv (2009), Roesch et al. (2007), Akam & Walton (2021), and Coddington et al. (2023). We highlighted that these theories are high-level mappings and may not be mechanistically accurate. We also added a footnote and forward reference to ensure this discussion is not overlooked.\n\n**The paper's comp neuro premise limits itself to the Forward-Forward algorithm, and ignores the host of other models in the field of biologically plausible learning.**\n\nOur main investigations revolve around an architecture inspired by the Forward-Forward algorithm. This does not mean we disregard other models in biologically plausible learning. Instead, our focus was to address a specific problem in biological reward-based learning: how neurons can coordinate learning to solve complex, nonlinear RL tasks using regionally homogenous, synchronously distributed error signals. While there are many related methods, they address different aspects of biologically plausible learning.\n\nTaking a reductionist approach allows us to tackle more complex modern RL tasks, highlighting what TD-learning with distributed error constraints can achieve relative to non-constrained current deep RL algorithms. This approach helps attribute performance differences to distributed errors and the downsides of our algorithm, without being confounded by other shortcomings in current biologically plausible deep learning.\n\nWe agree that integrating our model with other established methods, like DeepMind’s model of distributional dopamine coding (Dabney et al., 2020), is important. Since our last submission, we designed an extension of our algorithm based on the quantile regression method to learn distributions over values (Section 5). Our goal is to evaluate whether AD can learn such distributions, aligning with Dabney et al.’s work suggesting the brain’s value predictors learn distributions over values, rather than the mean value. Like Dabney et al., we use quantiles to reflect differences in optimism/pessimism across dopamine neurons. This distributional version of AD consistently achieved strong performance on DMC tasks, though it was slightly less sample efficient than the standard version of AD.\n\nWe have not yet integrated with Backpropamine (Miconi et al., 2020), which focuses on neuromodulated plasticity and is less directly relevant to our problem, but we we look forward to potentially exploring this direction in future work.\n\nFinally, we expanded our related work section (Appendix N) to include additional works in biologically plausible learning.\n\n**A single model consistent with data doesn’t preclude the existence of multiple other models, potentially with different mechanics, leaving the chance that the proposed model has little to do with the biological reality.**\n\nWe agree that our model solving the credit assignment problem in reward-based learning doesn't preclude other models from doing so. Our goal is not to claim our model is the only explanation, but to provide one potential solution to this problem, which, to our knowledge, has been lacking.\n\nTo our knowledge, no published works consistently solve all MinAtar tasks or our subset of DMC tasks without backpropagation; we are the first to do so. Ororbia & Mali (2022) proposed a similar backprop-free Q-learning algorithm, but their evaluations are limited to simpler tasks. We present results on 3 of these tasks in Appendix C (the fourth was not publicly available).\n\nWe also want to address the biological reality issue, and explain why we believe our work is biologically relevant.\nThe central hypothesis of our work is that synchronously distributed, locally homogeneous error signals, observed in the mesolimbic dopaminergic system, may already be sufficient to teach neurons to solve complex reward-based learning tasks. While we assume that NAc neurons encode approximations of the value function (and that VTA projections may compute errors specific to these approximations) in our experiments, we do not claim that our work proves this to be true; this assumption is part of our premise, not our hypothesis (we agree that alternative theories exist).\n\nRather, our findings suggest that if our assumptions hold, such conditions may be sufficient for complex reward-based learning by neural networks without the need for mechanisms explicitly coordinating credit assignment. This is counterintuitive and challenges established beliefs, potentially opening new avenues for explaining biological reward-based learning, and thus we believe it is a meaningful contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ftEcS4gXI1", "reviewer": "Reviewer_8Waq", "summary": "This paper addresses the distributed credit assignment problem in biological reinforcement learning, where a naive implementation of backpropagation is implausible. The authors show that it is possible to update action values using a variant of the forwward-forward algorithm specialized for RL. They then validate that this algorithm is effective on a range of standard benchmarks.", "questions": "- I was unsure whether the claim here is that the NAc is a multi-layer network or one layer in a multi-layer network.\n\n- If the authors would like their model to be taken seriously by neurobiologists, they need to do more to directly link it to empirical evidence.\n\n- Ideally the authors could make some neural predictions based on the model. It would also be helpful to understand how the predictions are different from those of other models.\n\nRebuttal update: I have raised my score from 6 to 7 following the authors' responses to my comments.", "limitations": "The authors extensively discuss limitations in section 6. There are no potential negative societal impacts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The problem (how to implement scalable deep RL in a biologically plausible way) is important.\n\n- The paper is clearly written, with a good review of both the biology and AI background.\n\n- The results are impressive given that the system does not use backprop.\n\n- I appreciated the careful ablation analysis.", "weaknesses": "- p. 5: The review by Glimcher (2011) is cited to support the claim that NAc (ventral striatum) neurons signal action value. In fact, there is a long tradition arguing that it is actually the dorsal striatal subregions (caudate and putamen) which signal action values, whereas ventral striatum signals state value (i.e., a form of actor-critic architecture; see Joel et al., 2002; O'Doherty et al., 2004). Most studies I'm aware of have shown that signals related to action value are relatively weak or absent in ventral striatum (see for example Ito & Doya, 2015).\n\n- Overall I felt that the biological evidence for the modeling framework was weak. What is the evidence that dopamine updates values locally in the NAc? This is physiologically tricky because dopamine acts postsynaptically through volume transmission, diffusing broadly in the extracellular space. What is the evidence for the particular form of hierarchical message passing architecture proposed here? Basically, I felt that the ideas here are interesting in principle but as presented are rather disconnected from empirical data.\n\nMinor:\n\n- p. 2: \"connection ,\" -> \"connection,\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful feedback, and we’re grateful for the opportunity to improve and clarify our work through answering your detailed questions and concerns.\n\n**There is a long tradition arguing that it is actually the dorsal striatal subregions (caudate and putamen) which signal action values, whereas ventral striatum signals state value.**\n\nWe acknowledge that there are multiple theories that provide different mappings of reinforcement learning frameworks to the brain’s reward circuitry. We want to clarify that our assumptions and findings are not in conflict with the actor-critic models by Joel et al. (2002) and O’Doherty et al. (2004). Actor-critic mappings of the reward circuitry, like Joel et al. (2002) and O’Doherty et al. (2004), argue the dorsal striatum is responsible for action selection, i.e. policy learning, whereas the ventral striatum is responsible for value learning. On the other hand, Q-learning is a simpler model that combines policy and value learning. The main difference (with respect to our work) is whether action selection occurs separate or together with value learning.\n\nAD does not make the argument that action selection need happen in the NAc. Rather, we just take the assumption that the NAc computes the Q-value function, which takes in the state and action as parameters, and performs Q-learning. The action can be provided from another region like the dorsal striatum. This view is consitent with the findings of works by Roesch et al. (2009), Mannella et al. (2013), and Aston et al. (2024).\n\nOn the other hand, we acknowledge that there exists research like Ito and Doya (2015) and Weglage et al. (2021) that suggest the competing theory that the dorsal striatum is responsible for signaling action values, whereas the ventral striatum signals state values. For completeness and openness, we will include this in the Limitations section.\n\n**The biological evidence for the modeling framework was weak. What is the evidence that dopamine updates values locally in the NAc, and for the particular form of hierarchical message passing architecture proposed here?**\n\nThe theory that dopamine updates values locally in the NAc is supported by Wightman et al. (2007), who found spatial and temporal heterogeneity of dopamine release within the NAc. This suggests that while dopamine concentration is homogeneous within each microenvironment containing a local subpopulation of NAc neurons, it can vary across different subpopulations.\n\nThe hierarchical message passing architecture is a design choice inherited from current deep learning practices, reflecting the passing of information between subpopulations of neurons. Our specific architecture is not intended to be a fully mechanistically accurate model of the NAc, but rather a computational model to demonstrate that it is possible to learn complex reward-based tasks using synchronously distributed, locally homogeneous error signals. The actual implementation in the NAc may differ. This counterintuitive and surprising finding potentially opens new avenues for explaining biological reward-based learning,  which is why we argue that we make a valuable contribution.\n\n**I was unsure whether the claim here is that the NAc is a multi-layer network or one layer in a multi-layer network.**\n\nEach layer in our model represents a local subpopulation of NAc neurons that receive the same signals from a subpopulation of dopamine neurons. However, as our model is not intended as a mechanistically accurate representation of the NAc, we do not claim that the NAc is a multi-layer network.\n\n**If the authors would like their model to be taken seriously by neurobiologists, they need to do more to directly link it to empirical evidence.**\n\nWe agree that additional biological connections can strengthen our work. However, the main contribution of our work is not to propose a biological model, but to show that synchronously distributed, locally homogeneous error signals observed in the mesolimbic dopaminergic system may be sufficient to teach neurons complex reward-based tasks. While we make some biological assumptions for modeling (e.g., NAc neurons encode approximations of the value function, and VTA projections may compute errors specific to these approximations), we do not claim our work proves these assumptions; they are part of our premise, not our hypothesis.\n\nWe demonstrate that if these assumptions hold true, such conditions may be sufficient for complex reward-based learning by neural networks, without the need for other mechanisms to explicitly coordinate credit assignment. Previously, it was believed that solving complex nonlinear tasks required neurons in different regions to sequentially pass error signals and explicitly coordinate their learning. This assumption is held by both biologically plausible local-learning algorithms and backpropagation. We show that this assumption may not be necessary.\n\n**Ideally the authors could make some neural predictions based on the model. It would also be helpful to understand how the predictions are different from those of other models.**\n\nWe agree that making neural predictions based on our model is an important direction. As an initial step towards better alignment with neurobiology, we performed experiments showing agreement with the distributional dopamine coding model by Dabney et al. (2025) in Section 5. We plan to further develop this aspect of our work in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c25P79nS6D", "reviewer": "Reviewer_dpw2", "summary": "In the paper, the authors propose a novel reinforcement learning algorithm. The algorithm is based on Q-learning and uses a  biologically inspired design to work with local error signals and updates, eliminating the need for the biological implausible backpropagation. The proposed algorithm, Artificial Dopamine, is motivated by the dopamine based learning system within the brain, concretely the ventral tegmental area (VTA) and the nucleus accumbens (NAc). It is realized with a novel recurrent layer/cell type, that predicates Q values from local error signals with an attention-like mechanism. The algorithm is evaluated against several baselines in a set of established RL benchmarks, and additional ablations add further insights on the algorithm.\n\n---\n**Post-Rebuttal/Discussion**: During discussion with the authors and fellow reviewers, open questions were tackled, and additional results were presented. I've increased my score to 8 due to this.", "questions": "The algorithm is based on Q-learning, could the main ideas be transferred to other RL backbones?\n\nThe reward needs to be globally available to all layers to compute the local prediction errors. Which implications does this global target induce? Any theoretical or practical drawbacks?\n\nHow crucial is the choice of the two activation functions in the AD cell?\n\nTypo (?) Line 101 mentions 15 tasks, while the rest of the paper mentions 14. Additionally, the main paper actually ‘only’ uses 10 tasks, the additional 4 are only provided in the Appendix for comparison with an additional baseline. Can this be clarified or better, unified in the main paper?", "limitations": "Limitations are discussed thoroughly in a separate Section. While detailed, they focus solely on the biological background, not mentioning technical/practical limitations.\nSome of such limitations, like only working with discrete/discretized action spaces and the difficulty in scaling to larger action spaces, are mentioned in other parts of the paper, it would be good to add such practical limitations and assumptions in the dedicated limitation section.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper tackles an interesting and important research direction with biological plausible reward-based learning, linking computer science advancements and neuroscientific findings nicely. The approach is motivated nicely, both through the background in neuroscience as well as with the connection to the recently proposed forward-forward algorithm. Similar to that, the proposed algorithm aims at training artificial neural networks without the biological implausible backpropagation.\n\nThe method is introduced and explained understandable, with details necessary to understand the algorithm conceptually as well as to implement it.\n\nThe included figures are of high quality and are helpful illustrations.\n\nThe presented evaluations are comprehensive, solid, and appropriate for the motivating research question. Especially the provided ablations answer questions occurring while reading the paper proactively.\n\nWhile not covering all aspects of biological plausible learning (as mentioned by the authors), the paper shows an elegant and competitive alternative to backpropagation that is grounded in neuroscientific insights on biological plausible reward-based learning.", "weaknesses": "While the majority of the paper is well written and understandable, one particular details is unclear and different sections seem to be contradicting. The AD cells/layers use multiple tanh layers, one for each action as stated and visualized. In Section 3.1, however, there is only one weight matrix W_att mentioned and used. Also in Figure 3 there is only the Q_t value, independent of an action. Inside each AD cell, are there multiple layers, one for each action, and if so, how are they used together for Q? Or is it one layer/matrix, with one dimension per action? (The first seems more plausible, but clarification and unification of notation and description would be beneficial.)\n\nWhile computational demanding and, hence, understandable, 10 runs are quite low for comparing RL algorithms (known problem in the community, although often neglected/ignored). Recent frameworks and metrics have been proposed tackled at these problems related to low number of runs and comparing overall algorithms performance [1]. It would be beneficial to add such metrics (e.g., performance profiles and IQM/IQR) to the paper for a better comparison of the proposed algorithm.\n\n------------\n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and detailed questions, and we’re glad to hear that you appreciate the novelty, clarity, and elegance of our work.\n\nTo address your specific questions and concerns:\n\n**Clarifications on Section 3.1.**\n\nWe understand how this may be confusing. Conceptually, the Q_t function takes two parameters, the state and the action, and outputs the value for that state-action pair. There exists multiple tanh layers, one for each action; W_att represents the weights of one of these layers.\nIn practice, for parallelization, like [1], the Q_t function is implemented to output a vector of |A| values (where A is the action space), i.e. one value for each possible action. Each tanh layer is used to compute the value of one corresponding action. We then take the argmax over that vector to determine the action to take. We see how Figure 3 is misleading; we will update it to show that the cell computes multiple Q_t values, one for each action.\n\n**10 runs are quite low for comparing RL algorithms; additional metrics like IQM/IQR would be beneficial.**\n\nWhile we agree that our results can be further bolstered with more runs, RL experiments are indeed computationally demanding (Appendix H), and like most researchers, we are constrained by available resources and environmental concerns. However, we hope that the number of tasks we evaluate our algorithm on helps alleviate some of your concerns; AD is evaluated on 14 standard RL tasks, with 10 runs on each task.\n\nFurther, we agree that additional metrics like IQM/IQR are beneficial for comparing the performance of RL algorithms. We’re working on additional figures that show the IQM and IQR of our data for our main results and ablation study, and will upload these before the end of the discussion period.\n\n**The algorithm is based on Q-learning, could the main ideas be transferred to other RL backbones?**\n\nYes, we believe they can, although we haven't yet experimented with this. We chose Q-learning as its one of the most supported hypotheses for how the brain performs reward-based learning, however the same learning principles should transfer to other RL paradigms like actor-critic models.\n\n**The reward needs to be globally available to all layers to compute the local prediction errors. Which implications does this global target induce? Any theoretical or practical drawbacks?**\n\nJust making the reward signal globally available to all layers does not incur any notable drawbacks we are aware of; there is evidence that this occurs in biological learning (Section 2.1), and providing such a signal to each layer does not add any significant computational costs for modern hardware (in fact, it improves parallelization, as each layer’s update can be computed in parallel, Section 3).\n\nHowever, there is a significant drawback to replacing the sequentially propagated error signal with just per-layer reward and prediction errors, namely the layers can no longer directly coordinate their learning. Specifically, the upper layers of the network can't influence the learned representations of the lower layers via sequentially propagated gradients. The problem with gradient propagation is that there is no evidence it in the brain, despite its known importance in deep learning. This is partially why we believe our results are surprising and significant: we show that just distributed, local prediction errors and updates may already be sufficient to learn many complex reward-based learning tasks, without the need to directly coordinate updates across layers during learning.\n\n**How crucial is the choice of the two activation functions in the AD cell?**\n\nWe experimented with alternative choices for these two activation functions, including replacing the relu function with a leaky relu, and replacing the tanh function with a sigmoid and with a second relu function. Most combinations of different activations did not have a significant impact on performance, with the exception of using two relu functions, which made learning unstable.\n\nFor transparency, our experiments on activation function choice were not extensive; there could certainly be combinations of other activation functions that lead to better performance.\nHowever, as our primary goal is not to achieve state-of-the-art performance, we believe that any potentially unrealized gains in performance does not interfere with our contributions, and thus leave further hyperparameter tuning of AD cells to future work.\n\n**Line 101 mentions 15 tasks, while the rest of the paper mentions 14; the main paper uses 10 tasks.**\n\nWe agree this can be further clarified. Line 101 is indeed a typo; we evaluated AD on 14 tasks, including 5 tasks from the MinAtar testbed, 5 tasks from the DeepMind Control Suite, and 4 classic control tasks. We relegated the 4 classic control tasks to Appendix C because they are relatively less challenging compared to the MinAtar and DMC tasks (despite still being an informative starting point for evaluating the performance of reward-based learning algorithms). As we achieved consistently strong performance on each of these 4 tasks, to avoid cluttering the main paper body (given the space limit), we moved them to the appendix.\n\nTo address this issue, we have fixed the typo, and updated the Experiments section (Section 4) to explicitly state the number of tasks.\n\n**While technical and practical limitations are mentioned in other sections, the Limitations section only discusses limitations related to biology.**\n\nWe will add a dedicated paragraph in the Limitations section to discuss technical and practical limitations, including discretized action spaces, current difficulties with scaling, and the number of runs per task.\n\n---\n**References**\n\n[1] Mnih, Volodymyr, et al. \"Human-level control through deep reinforcement learning.\" *nature* 518.7540 (2015): 529-533.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is broad agreement amongst the reviewers that the biologically-inspired algorithmic contribution of the paper is interesting, novel. and significant, both because it demonstrates an RL method without backpropagation and because it builds upon constraints from hypotheses in neuroscience. The main points of contention concern the extent to which the paper presents unsettled neuroscientific hypotheses as fact and the extent to which the paper offers or claims to offer insight regarding biological systems. Especially after the discussion period, most of the reviewers agree that the paper is sufficiently clear about its intent and its claims though there is some disagreement on this point.

The overall assessment seems to be that the paper is interesting on a technical/conceptual level but that the paper could be stronger in its communication with the neuroscience community. I recommend acceptance, but I also urge the authors to carefully consider the critiques from these reviewers and, while preparing the camera-ready version, to focus on clear, up-front statements about the disputed nature of the underlying neuroscience hypotheses and about the limited nature of any biological insights.
Decision: Accept (poster)