Temporal-Difference Learning Using Distributed Error Signals

Authors: jonas  guan
shon eduard verch
claas  voelcker
ethan c jackson
nicolas  papernot
william a cunningham
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:48:26

Abstract: A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc). Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.

Main paper: 
 introduction: 
 Computer science and neuroscience have enjoyed a longstanding and mutually beneficial relationship. This synergy is exemplified by the inception of artificial neural networks, which drew inspiration from biological neural networks. Neuroscience also adopted temporal-difference (TD) learning [62] from reinforcement learning (RL) as a framework for biological reward-based learning in the midbrain [23, 59] . At the intersection of these ideas, deep RL has much benefited from and contributed to interdisciplinary progress between the two fields [45, 11] .An interesting problem raised in biological learning is how signals transmitted by the neuromodulator dopamine computationally induce coordinated reward-based learning. In the mesolimbic system, dopamine is synthesized by dopamine neurons in the ventral tegmental area (VTA) and transmitted through the mesolimbic pathway to several regions, including the nucleus accumbens (NAc). There, it is synchronously distributed in regionally homogeneous concentrations [58] , and serves as a reward prediction error signal for synaptic adjustments via TD learning [9, 23] . 1 Figure 1 shows a conceptual illustration: the medium spiny neurons in the NAc receive error signals distributed locally in their region via dopamine. Computationally, however, this theory faces the credit assignment problem [25] : the individual synaptic updates using just local errors must somehow work in coordination to improve the collective prediction. 2 Are distributed error signals alone sufficient to coordinate neurons to learn complex reward-based learning tasks? Deep RL typically solves the credit assignment problem using backpropagation (BP) [57] . BP propagates the global error backwards through the network, and computes the gradient (w.r.t. the global error) of each layer's synaptic weights sequentially via the chain rule. In contrast to the synchronously distributed errors in the NAc, BP involves neurons sequentially communicating error signals with each other. This sequential propagation explicitly coordinates learning, but also creates dependencies: each layer's updates depend on the error of subsequent layers. This is known as the update locking problem, which is biologically implausible [52] , limits parallelization, and cannot explain how distributed error signals may support coordinated learning.Recent ML research on more biologically plausible alternatives to BP may offer critical insights. PEPITA [21] and Forward-Forward (FF) [30] both replace BP's backward learning pass with a second forward pass to address update locking. Most relevantly, Hinton [30] made a surprising discovery: layers can learn useful representations for subsequent layers even when trained independently of the errors of those subsequent layers. In FF, each layer generates its own prediction and error, and is only trained to learn hidden representations that minimize the local error. The subsequent layer takes these representations as input, and achieves better performance over training, despite being unable to send errors to the previous layer. This improves the collective global prediction without explicit, sequential coordination of error signals. To the best of our knowledge, these learning principles have not been explored in RL or biological reward-based learning.Drawing a novel connection, we hypothesize that the computational mechanisms that enable FF's independent, per-layer training may also enable distributed error signals to support coordinated rewardbased learning. To test our hypothesis, we design ARTIFICIAL DOPAMINE (AD), a new deep Qlearning algorithm that trains RL agents using only synchronously distributed, per-layer TD errors, and evaluate its performance on a range of discrete and continuous RL tasks. This provides a potential explanation for credit assignment in NAc dopaminergic learning at the algorithmic level of analysis [42] . Our results show that AD can solve many common RL tasks often as well as deep RL algorithms that use backpropagation, despite not propagating error signals between layers. Thus, we computationally demonstrate that distributed errors alone may be sufficient for coordinated reward-based learning.AD networks inherit several ideas from FF, which differ from traditional neural networks in two significant ways. First, each layer in an AD network computes its own prediction and receives a corresponding error (Section 3.1). This per-layer error mirrors the locally homogenous distribution of dopamine, and the computation of error and updates can be synchronously parallelized across layers; there are no dependencies across layers. Second, we use forward 3 connections in time to send activations from upper to lower layers (Section 3.2). This provides an information pathway for upper layers to communicate with lower layers using activations, rather than error signals, and empirically improves performance. Figure 2 outlines our architecture, unfolded in time.The AD cell (Figure 3 ) is where we differ most significantly from FF. Its role is to compute the local Q-prediction and TD error. FF is designed to separate real from fake (generated) data, a binary classification task. But the NAc is theorized to predict value, a regression task, and therefore needs more precision. To achieve this, we introduce an attention-like mechanism for non-linear regression without using error propagation (Section 3.1).We evaluate AD on 14 discrete and continuous RL tasks from the MinAtar testbed [70] , the DeepMind Control Suite (DMC) [64] , and classic control environments implemented in Gymnasium [65] . t represents the activations of layer l at time t, and s t the input state. The blocks are AD cells, as shown in Figure 3 . Similar to how dopamine neurons compute and distribute error used by a local region, each cell computes its own local TD error used by its updates; errors do not propagate across layers. To relay information, upper layers send activations to lower layers in the next timestep. For example, red shows all active connections at t = 1.MinAtar tasks are miniaturized versions of Atari games, and DMC contains continuous control tasks with simulated physics. These environments are complex enough to reflect many challenges in modern RL [14] , yet remain adequately tractable so as not to necessitate extra components like convolutional layers, which may be confounding when attributing performance. We benchmark AD against DQN [45] , SAC [26] , and TD-MPC2 [28] baselines, and conduct ablation studies to examine the effects of the forward connections and additional layers. Our results in Figures 4 and 8 show that AD learns to solve many of these tasks with comparable performance to the baselines, using just per-layer TD errors. Our code is available at https://github.com/social-ai-uoft/ad-paper.To summarize our core contributions:• Are distributed TD error signals sufficient to solve credit assignment and learn complex reward-based learning tasks? We construct a computational example of an RL agent that learns using only distributed, per-layer TD errors. This provides evidence that dopamine-distributed signals alone may be enough to support reward-based learning in the nucleus accumbens.• We design a Q-learning algorithm, ARTIFICIAL DOPAMINE, to train our agent. Like Forward-Forward, AD does not propagate error signals across layers. Unlike FF, we introduce a novel cell architecture to compute Q-value predictions, as Q-learning is a regression task.• We evaluate our agent on 14 common RL tasks in discrete and continuous control, and show that AD can often achieve comparable performance to deep RL algorithms, without backpropagation. 
background: 
 2.1 Reward-Based Learning in the NAc4 The nucleus accumbens (NAc) plays a critical role in the mesolimbic reward system, and is theorized to predict action value [41] . The predicted action value is encoded via the firing rate of NAc neurons [23] , which reach dopamine neurons in the ventral tegmental area (VTA) via projections and influence their activity [60] . In addition, the VTA receives reward signals from sensory inputs, such as the detection of sugar on the tongue [23] . The value predictions and reward signals enable the VTA to compute reward prediction errors, i.e. differences between expected and actual reward. These reward prediction errors (more specifically TD errors) are encoded via dopamine, and projected through the mesolimbic pathway to regions of the ventral striatum, including the NAc, and lead to synaptic adjustments [60] . When there is a positive reward prediction error, the average activity of dopamine neurons increases; when there is a negative error, the average activity decreases [9] .The exchange of value predictions and error signals between the NAc and VTA sets the stage for TD learning. TD models have shown strong congruency with observed dopamine activity [48] , and are widely established as the primary theory for mesolimbic dopaminergic learning.What is less known is how NAc neurons computationally use the dopamine-encoded error signals to coordinate learning. Dopamine-encoded error signals are distributed [58] , which makes credit assignment difficult, and there are no other known mechanisms that NAc neurons utilize to communicate error signals for explicit coordination. We illustrate the distribution process in Figure 1 . Each dopamine neuron projects dopamine along its axon through the mesolimbic pathway, then synaptically releases it in synchronous bursts to the immediate juxtasynaptic area. This causes dopamine concentration in the region to peak, activating dopamine receptors in NAc medium spiny neurons. Dopamine concentration levels are locally homogeneous near the synapses [58] , but can vary across different regions of the NAc [68] , as dopamine neurons need not all fire at once. This supports the distribution of localized error signals. While Figure 1 is clearly not to scale, dopamine neurons are indeed significantly larger than medium spiny neurons. Their large cell body size enables them to support large terminal fields [13] , allowing each dopamine neuron to widely distribute its signal to groups of NAc neurons.Temporal-difference learning (TD learning) methods are a family of RL approaches that aim to solve the problem of estimating the recursively defined Q function [62] . In their most basic form, TD learning methods can be implemented as lookup tables, where an estimate of the Q function is kept for each state-action pair. However, in many practical applications with large or continuous state spaces, a tabular representation of the Q values for all distinct state-action pairs is computationally infeasible. In these cases, function approximation, for example via linear regression or neural networks, is necessary to obtain an estimate of the Q function. These methods, called Fitted Q Iteration [4] or Deep Q Learning [44] , use the squared temporal difference error as a regression loss L(s, a, s ′ ) = δ(s, a, s ′ ) 2 , where δ is the TD error, and update a parametric Q function approximation via gradient descent. To prevent the double sampling bias and other instabilities, only Q(s, a) is updated and the next states value is estimated with an independent copy Q that is not updated. This is commonly called the bootstrapped Q loss. Given parameters θ, this loss can be written as:L(s, a, s ′ , θ) = Q θ (s, a) -r(s, a) + γ max a ′ Qθ (s ′ , a ′ ) 2For a more formal description of our TD learning setting, see Appendix A.The Forward-Forward (FF) algorithm [30] is a greedy multi-layer learning algorithm that replaces the forward and backward passes of backpropagation with two identical forward passes, positive and negative. The positive pass is run on real data; the negative on fake (generated) data. The goal of the model is to learn to separate real from fake data. During training, each layer of the network performs this classification independently using a measure called goodness, which computationally acts as the logit for this binary classification task. Each layer computes its own per-layer error for updating. FF's de facto measure of a layer's goodness g is the sum of its squared hidden activations h i minus some threshold τ , i.e. g = i h 2 i -τ . However, due to the simple goodness formula, if each layer directly passes its activations to the next, the next layer often trivially learns the identity function. It then uses the sum of its last layer's hidden activations as its goodness. This traps the layer in a local optimum. To avoid this, FF uses layer normalization [5] to normalize the activations before passing them on, which keeps the relative values of the activations, but makes them sum to 0.Compared to BP, a global algorithm that requires the entire network to be updated sequentially, FF is a local algorithm that updates each layer independently via local, layer-wise losses. Critically, the authors of [30] showed that layers can learn useful representations that help subsequent layers even though they are trained on local errors, without explicit credit assignment. However, for a feedforward architecture, one glaring limitation is that later layers cannot relay any information to earlier ones. Hinton suggests addressing this with a multi-layer recurrent architecture, where for each layer l in the network, the input is determined by output of the layer l -1 and that of l and l + 1 at the previous time step t -1. This allows for top-to-bottom information flow through the network via activations, which is more biologically plausible. t is the activations of the cell l at time t, andQ[l]t is a vector of Q-value predictions given the current state and each action. We compute the cell's activations h[l] t using a ReLU weight layer, then use an attention-like mechanism to computeQ[l] t . Specifically, we obtain Q[l]t by having the cell's tanh weight layers, one for each action, compute attention weights that are then applied to h[l] t . Each cell computes its own error. 
artificial dopamine: 
 ARTIFICIAL DOPAMINE (AD) is a deep Q-learning algorithm that trains deep RL agents using distributed, per-layer TD errors. AD inherits the per-layer predictions and local computation of errors from FF, and adopts the forward-in-time downward connections from FF's recurrent variant. Informally, the intuition is that the per-layer error acts similarly to the locally homogeneous distribution of dopamine in the NAc. The per-layer errors can be computed in a parallelized, distributed fashion; each layer computes its own local error, which the neurons of the layer use to adjust weights according to their contributions to the error. Similarly, the NAc neurons near the synapses of each dopamine neuron receive the same error signal (encoded via dopamine), which they use to adjust their synaptic weights, according to their previous activity shortly before receiving the error. During inference, the network uses the average Q-values across the layers to produce a final prediction.Since the NAc is theorized to predict action value [23] , i.e. Q-value, our prediction task deviates from that of FF, which performs binary classification. In general, predicting action value is a nonlinear regression task. To learn this task using a neural network, without resorting to the biologically implausible BP, we design an attention-like mechanism that learns sets of "attention" weights--one set per action head, which introduces nonlinearity. We encapsulate this process in AD cells.Our network architecture is composed of layers of AD cells, each of which makes its own Q-value prediction, computes its own local TD error, and updates its own weights. At inference, the final prediction of the network is the average of each cell's Q-value predictions. We use an attention-like mechanism [6] that learns a weighted sum of the cell's hidden activations to predict the Q-value. The hidden activations are passed to other cells, but the attention weights and Q-value are withheld. This mechanism simply serves to functionally simulate the complex nonlinear capabilities of biological neurons [10] ; we are not attempting to draw an analog between our mechanism and any biological counterpart, and design choices are primarily made based on empirical performance.The concept of a cell is reminiscent of the design of a recurrent neural network. In our case, there is a single cell per layer, so we use the terms somewhat interchangeably for clarity of exposition. The vital difference is that no loss information is propagated between cells via BP; that is, there is no BP through time. Instead, the same BP are passed to the cell above (i.e., hidden layer) at the same timestep and to the layer below at the next timestep. In the absence of BP, these activations provide a pathway for upper cells to communicate information to lower cells (see Figure 2 ). We discuss in more detail how these connections operate in Section 3.2.Figure 3 presents the cell design. Each cell is mechanistically identical and takes in two inputs: the hidden activations of the cell below at the current timestep (or observation, if lowest cell), and the hidden activations of the cell above at the previous timestep. It produces two identical outputs, sent to the cell above immediately, and the cell below at the next time step. At the start of an episode, the activations from the previous timestep are zeroes. The top cell only receives one input and produces one output.The attention-like mechanism for Q-value prediction works as follows. Each cell computes its hidden activation, h[l] t , using the layer above's hidden activations from the previous timestep, h[l+1] t-1 , and the layer below's hidden activations at the current timestep, h . Specifically, it passes the concatenation of h t . The ReLU weight layer multiplies the concatenated input [h[l-1] t , h [l+1]t-1 ] by its learned weight matrix W [l] , then applies the ReLU nonlinearity function. In parallel, the cell also uses [h[l-1] t , h [l+1]t-1 ] to compute "attention" weights, by multiplying it with the learned weight matrix W[l] att then applying the tanh function. Finally, the cell takes the dot product between the output of the tanh layers, tanh(W[l] att • [h [l-1] t , h [l+1] t-1 ]), and the hidden activations h[l] t , to compute the cell's Q-value prediction Q[l] (s t , a). Each cell reuses its internal weights over time; for example the matrix W [1] in is used at each timestep in the first cell. Therefore, the full computation performed by a cell is:h [0] t = s t , h [L+1] t = 0, h [l] t-1 = 0 h [l] t = relu W [l] in • h [l-1] t , h [l+1] t-1 Q[l] (s t , a) = tanh W [l] att • h [l-1] t , h [l+1] t-1 ⊺ h [l] tOptionally, like [30] , our architecture supports skip connections between cells. In such cases, the additional input from a skip connection is simply concatenated with [h[l-1] t , h [l+1] t-1 ] before computing h [l] t .As shown in Figure 2 , each cell passes its state to the cell l + 1 above at the current timestep t, and to the cell l -1 below in the next timestep t + 1. The information flow is strictly unidirectional to match the direction of time flow in RL environments. This is necessary as interacting with the environment happens sequentially, meaning future information will not be available when acting.Although we do not backpropagate gradients across cells, information does flow from upper layers to lower layers via the temporal connection (forward in time). The upper layers use the connections to communicate with lower layers via activations, which is more biologically plausible [49] . Our results in Figure 4 suggest that these connections can greatly increase network performance in the absence of BP. The intuition for adopting these forward-in-time connections is that they are well-suited to take advantage of the temporal structure of the Q-values of trajectories for better learning. Given a good policy, the Q-value predictions of a well-trained model should remain stable through each state of a trajectory (assuming the dynamics are reasonably deterministic). This means that the Q-value prediction of the current timestep, and the hidden activations used to make this prediction, can often still be useful for predicting the Q-value of the next timestep. In contrast, in FF's experiments on image classification, this effect is forced-FF repeats the same input every timestep, reducing computational efficiency. Our results empirically support the effectiveness of forward connections for Q-learning, particularly in more complex environments. 
experiments: 
 The main goal of our experiments is to evaluate whether distributed, per-layer TD errors are sufficient for learning complex RL tasks that are typically only solved with backpropagation and sequential errors. Our criterion of sufficiency is how well the agent learns to solve the given task, measured in terms of average episodic return. Since there are no official standards that define solving these environments, we use the performance of established RL algorithms (i.e. DQN, TD-MPC2, and SAC) as the gold standard to compare against. These algorithms are commonly used to solve the environments we choose and are known to be strong performers [70, 28, 26] .Like Hinton [30] , our aim is to investigate the learning capabilities of a novel algorithm that operates under additional biological constraints, rather than pursue state-of-the-art performance. Thus, we opt for a simple implementation with few extensions. The only extensions we employ are experience replay [45] and the Double-Q learning trick [66] -which are standard for deep Q-learning-and skip connections from the input to upper layers for the DMC environments. We do not use convolutional layers, as these more closely resemble the brain's visual cortex [40] , and similar structures are not found in the mesolimbic system. We implement our algorithm in Jax [12] .Training Process. The training process of our RL agent is based on the standard DQN training algorithm with experience replay [45] . During training, we use a randomly sampled replay buffer that stores every transition observed from the environment. We replay sequences of short length, similar to how other recurrent architectures are trained [29, 35] , and compute the local updates for each cell sequentially according to the network graph. Since the local updates are distributed and per-layer, they can be computed in parallel. We provide a formal description of our training process in Appendix B.Environments. We run our experiments on 2 RL benchmarks and 4 classic control tasks, totaling 14 tasks. MinAtar [70] is a simplified implementation of 5 Atari 2600 games: Seaquest, Breakout, Asterix, Freeway, and Space Invaders. The DeepMind Control (DMC) Suite [64] is a set of low-level robotics control environments, with continuous state spaces and tasks of varying difficulty. For our experiments, we used a discretized action space, following Seyde et al. [61] , as our architecture is currently only developed for discrete Q-learning. From the DMC tasks we select Walker Walk, Walker Run, Hopper Hop, Cheetah Run, and Reacher Hard. In addition, we provide results on the classic control tasks Cart Pole, Mountain Car, Lunar Lander, and Acrobot, which we include in Appendix C. For a more elaborate discussion on these environments and our task choice, see Appendix I.Baselines. On MinAtar, we compare our results against a fully-connected DQN to make performance comparisons more direct and informative. As the original baselines presented in Young and Tian [70] used a CNN, we replaced the CNN with fully connected layers and tuned hyperparameters for fairness of comparison. We find that the new architecture performs as well as or better than the one presented by Young and Tian [70] . Specifically, we use a double DQN with 3 hidden layers of 1024, 512, and 512 units with ReLU activations. This baseline achieves strong performance and has a number of trainable parameters comparable to our networks.For the continuous control tasks, we show that our method almost reaches the performance of state-of-the-art algorithmic approaches such as SAC [27] and TD-MPC2 [28] , which rely on backpropagation. The results were taken from Yarats and Kostrikov [69] and Hansen et al. [28] respectively. We do not change the underlying architectures or hyperparameters.Network architecture and hyperparameters. On MinAtar, we use a 3-hidden-layer network with forward activation connections. The cell output sizes are 400, 200, and 200. Due to additional connections within cells and from upper to lower cells, this architecture has a similar number of trainable parameters as the DQN. On DMC, we use a smaller network with cell output sizes 128, 96, and 96, and discretize the action space following Seyde et al. [61] . For more details, see Appendix G. For each benchmark, we use the same network and hyperparameters across all tasks to test the robustness of our architecture and learning algorithm. 
results: 
 We present the results of AD on MinAtar and DMC environments in Figure 4 , and compare its performance against DQN, SAC, and TD-MPC2. Figure 4 shows the mean episodic return over episodes across 10 random seeds, with standard error. We also provide 95% bootstrap confidence intervals [55] in Appendix D and aggregate statistics in Appendix E. We additionally perform ablation studies by removing the forward connections to lower layers, and measuring the performance of a single-layer AD cell. We show both the forward connections and multiple layers contribute to performance (Figures 5 and 6 ). Finally, we evaluate an implementation of AD that learns distributions over Q-values, based on recent work by Dabney et al. [20] , and find that AD shows promising results (Figure 7 ).Comparison against baselines. We find that AD is able to learn stable policies reliably in all test environments. On MinAtar tasks, our agent achieves comparable performance to DQN on Breakout, and slightly surpasses DQN's performance on Asterix and Freeway, while DQN performs better on Seaquest and Space Invaders. On all evaluated DMC tasks, we see that our method's results are close or on par with those of SAC and TD-MPC2 when using both backward connections and multiple layers, however, sample efficiency is marginally lower.We note that the action space discretization we utilize for DMC tasks may complicate the comparison, as it both slows down our training but can benefit algorithms (compare Seyde et al. [61] ). In addition, we do not show results on the hardest DMC tasks because of the difficulties in scaling our approach to large action spaces, which leads to rapid growth in network parameters.Overall, our results demonstrate that AD shows biological distributed error signals may allow for coordinated learning in several RL domains. Therefore, further refining and improving the architecture for state-of-the-art RL benchmark performance may be an exciting and promising direction for future work.Forward connections to lower layers. To further measure the performance impact of the forward connections in time, we compared the temporal forward version of the network to a 3-layer AD network without the forward-in-time connections. All other hyperparameters are the same as the 3-layer AD network. As shown in Figure 5 , this resulted in a moderate drop in performance in most environments, increased variance in the training, and devastating drops in performance on Seaquest and Asterix. These tasks are the most complex out of the five. We provide additional discussion on AD's performance in Seaquest, where the performance difference is most significant, in Appendix N. These results suggests that the information channels from the upper to lower layers are vital for performance on many tasks. On this same note, AD's ability to achieve similar performance to DQN when the forward connections are added suggest that forward connections may be an effective replacement for backpropagation in certain tasks.Single-layer performance. Another important question about our proposed architecture is whether the cells learn to coordinate with each other. A concern is that the majority of the learning may be accomplished by the lowest cell, If the performance of the multi-layer AD network does not improve over the single cell, it would suggest that we cannot explain AD's performance as a result of the cells coordinating their learning using distributed errors. We show in Figure 5 that the multi-layer version of AD outperforms the single-layer at all tasks, and the single-layer cell fails at Seaquest and Asterix. An additional concern along these same lines is whether a wider, single-layer AD cell may achieve the same level of performance as multi-layer AD. In Figure 6 , we show that increasing the layer size of a single-layer AD cell does not result in clear increases in performance in DMC tasks. We also experimented with increasing the layer size of a single-layer cell for Seaquest and Asterix from 400 to 600 and 800, and did not find noticeable improvements in either case.Distributional RL. Recent work by Dabney et al. [20] suggest that the brain's value predictors may be learning distributions over future values, rather than just the mean, as previously believed. Dabney et al. [20] argue that different dopamine neurons in the VTA may have different scalings for positive and negative reward prediction errors-intuitively, they can be more optimistic or pessimisticwhich results in the predictors learning a distribution over the values. Interestingly, this coincides with the development of distributional RL, whose algorithms aim to learn such distributions.To better align our work with the findings of Dabney et al. [20] , we additionally implement a version of AD that learns distributions over values, and evaluate it on the DMC tasks. Our implementation is based on Quantile Regression (QR) DQNs [19] , and requires just a simple modification to each AD cell. Rather than predicting a single Q-value, each cell predicts 10 Q-values that each match to one quantile of a 10-quantile QR-DQN. The tradeoff is that this requires additional compute.Our results in Figure 7 suggest that AD may be well-suited for distributional learning. In each of the tasks, our agent achieves similar performance to the standard version of AD, and only slightly lags behind on Hopper Hop. This may be a result of the greater sparsity of the Hopper environment, which makes it more difficult for distributional RL algorithms to learn. 
limitations: 
 We proposed an algorithm that trains an RL agent using only distributed TD errors, which provides a computational example of how distributed error signals may be sufficient to solve credit assignment in reward-based learning in the NAc. However, our model does not accurately capture all aspects of the relevant biology. In our model, we use per-layer TD errors as an analogy for dopamine neurons distributing error signals to local regions around their synapses. But unlike in artificial neural networks, which form the basis of our architecture, neurons in the NAc are not clearly organized into layers. AD's hierarchical message passing architecture is a design choice we inherit from deep learning practices, and not meant as a mechanistic model of the NAc. Furthermore, activations in biological neurons are communicated asynchronously, and can form recurrent loops, which we do not account for.In addition, we make some assumptions regarding biological reward-based learning that are not yet conclusive in neuroscience. Most importantly, we assume that neurons in the NAc learn to output action values, and dopamine neurons in the VTA receive reward signals and the predicted action values to compute and return TD errors. 5 While these assumptions are widely supported by research [23, 41, 46, 47] , there exist other theories and empirical results that provide alternative explanations for mesolimbic reward-based learning. Some of these works include Roesch et al. [56] , who suggest that on-policy value-based learning (SARSA) better explains dopamine activity than off-policy value-based learning (e.g. Q-learning); Takahashi [63] and Chen and Goldberg [16] , who provide evidence that map subregions of the striatum to actor-critic models; Ito and Doya [32] and Weglage et al. [67] who suggest the dorsal striatum signals action values, whereas the ventral striatum signals state values; and Akam and Walton [2] and Coddington et al. [17] , who respectively propose how dopamine may be used by the brain to perform model-based and policy-based learning rather than just value-based learning. Furthermore, while there is strong evidence that some form of TD learning is used by the brain, mappings between RL frameworks and dopaminergic learning may not be mechanistically accurate even if they are behaviorally accurate.There are also several technical limitations of our work. First, like other Q-learning algorithms, AD requires discrete action spaces. To solve the DMC tasks, which have continuous action spaces, we discretized the action space, following the method used by Seyde et al. [61] (Section 4). While this is consistent with other work in RL, it introduces additional complexities [61] and may not reflect biology. Second, within each AD cell, the number of tanh weight layers grows in proportion to the size of the action space (Figure 3 ). This limits the scalability of AD for tasks with large action spaces. We can mitigate the effects of this issue using a matrix decomposition trick described in Appendix F, but AD currently cannot scale to very large action space DMC tasks like Humanoid or Dog [64] . Third, computational constraints limited the number of runs we perform per task. Additional experiments can further improve the robustness and generalizability of our results. Finally, our work isolates one system of biological learning, and attempts to provide a computational explanation without accounting for other systems of learning, for example hippocampal contributions to value-based learning [7] . But the brain's learning capabilities are likely a result of combining signals from several systems, which may not be divisible [50] , and the NAc may be just one part of a larger value-based system [8] . Unlike backpropagation, the brain utilizes multiple types of learning in conjunction, both supervised and unsupervised, using local and global signals. AD only models one type of learning, i.e. error-driven learning using distributed reward signals to induce local updates. Other biologically plausible algorithms, such as ANGC [53] , may provide explanations for other forms of learning, which may be critical to building a more complete understanding of biological rewardbased learning. Indeed, aspects of Hebbian learning or active inference are likely critical to achieving a fully biologically plausible, efficient, and powerful learning system. In that light, we explore learning with just distributed error signals not to rule out the importance of other methods, but to demonstrate that this one principle alone may be sufficient to solve some complex RL tasks nearly as well as BP. We believe that a key to achieving general, human-like intelligence will be the integration of these different learning methods and learning signals; this is an exciting direction we aim to explore in future work.We consider a standard discounted infinite horizon Markov Decision Process (MDP) setting with states S, actions A, a transition kernel p(s ′ |s, a), a reward function r : S × A → R and a discount factor γ ∈ [0, 1]. This is a typical setting for reinforcement learning [62] .The goal of TD learning is to obtain a policy π(a|s) that maximizes the discounted future sum of rewards. The value function Q : S × A → R measures how valuable a given action a ∈ A is in a given state s ∈ S, and can be used to directly compute a policy. It is defined via a recursive formula Q(s, a) = r(s, a) + γ s ′ p(s ′ |s, a) a ′ π(a ′ |s ′ )Q(s ′ , a ′ ). When experience in the form of s, a, r, s ′ tuples becomes available from interaction with the MDP, the Q function estimates are updated using the TD error δ(s, a, s ′ ) = r(s, a)+ γ a ′ π(a ′ |s ′ )Q(s ′ , a ′ ) -Q(s, a). The update rule is Q k+1 (s, a) ← Q k (s, a) + α k δ(s, a, s ′ ),where α specifies the learning rate.Algorithm 1 AD Q-Learning Initialize replay memory D to capacity N Initialize action-value functions Q [1] , . . . , Q [L] with random weights θ [1] , . . . , θ [L] Initialize target functionQ[1] , . . . , Q[L] with weights θ [i] ′ = θ [i] for every i ∈ {1, . . . , L} for episode 1, M do Initialize s 1 = initial state for t = 1, T do With probability ε select a random action a t otherwise select a t ∈ arg max a 1 L L l=1 Q [l] (s t , a; θ [l]) Execute action a t in emulator and observe reward r t+1 and next state s t+1 Store experience (s t , a t , r t+1 , s t+1 ) in D Sample minibatch of K-sized episodic sub-trajectories (s j-K , a j-K , r j-K+1 , s j-K+1 ), . . . , (s j , a j , r j+1 , s j+1 ) from D with uniform randomness Replay states s j-K , . . . , s j-1 through online network to obtain layer activations h[1] j-1 , . . . , h [L] j-1 Set current layer input x ← s j for layer l = 1, . . . , L do Set TD target Y [l] j = r j+1 if episode terminates at step j + 1 r j+1 + γ max a Q[l] s j+1 , Q [l] (s j+1 , a; θ [l] ); θ [l] ′ otherwise Set prediction y j ← Q [l] (s t , a j ; θ [l]) using x and last activations h[l-1] j and h [l+1] j-1Perform a gradient descent step on (Y[l] j -y j ) 2 with respect to the layer parameters θ[l] Set input for next layer x ← h [ℓ] j (x) end for Every C steps reset θ [i] ′ = θ [i]for every i ∈ {1, . . . , L} end for end forBeyond MinAtar and DMC, we also evaluate AD on 4 classic control environments: Cart Pole (also known as inverted pendulum), Mountain Car, Lunar Lander, and Acrobot, and compare its performance against DQN. We show our results in Figure 8 .The main purpose of these experiments is to further support AD's robustness across different RL environments, and indirectly compare its performance against Active Neural Generative Coding (ANGC) [53] , another biologically-inspired deep RL algorithm that provides a more plausible In addition, we implement 2 DQNs with different hyperparameters as baselines for comparison: first is CleanRL's reference DQN for classic control problems [31] , and second is the DQN tuned by [53] for comparison against their ANGC agent, which we refer to as ANGC's DQN. The CleanRL DQN provides a well-established, publicly-vetted baseline, whereas ANGC's DQN serves as a reference point for us to indirectly compare AD's performance with ANGC.To better demonstrate AD's robustness across environments, we use the same set of hyperparameters for each of the environments. Specifically, we use a 2-layer AD network, with output sizes 128 and 96, a fixed learning rate of 2.5e -4 with Adam optimization, an exploration fraction of 0.2, and final epsilon of 0.05. All other hyperparameters are the same as our network for MinAtar, shown in Appendix G. As neither CleanRL nor ANGC's DQN uses a learning rate scheduler, we also removed ours for better comparison. CleanRL's DQN is a 2-layer network with 120 and 84 units; the same DQN is used for all 4 environments. ANGC uses different 2-layer DQNs for each of their environments, which they tuned for environment-specific performance. Their Cart Pole DQN has 256, 256 hidden units; Mountain Car has 256, 192, and Lunar Lander 512, 384. For Acrobot, we used the same hyperparameters as the Cart Pole DQN, given the similarity of the environments, which worked well. For more hyperparameter details, we refer the reader to [31] and [53] ; both works present excellent detail.We made two adjustments when reproducing CleanRL and ANGC's DQNs. First, since AD utilizes double Q-learning to improve its learning stability, for better comparison, we also enhanced both DQNs with double Q-learning, given that it is a simple enhancement without additional computational costs. Second, specific to ANGC's Cart Pole DQN, we found that the agent's learning has very high variance with the given hyperparameters, and the agent does not consistently achieve high performance. To counter this, we decreased the learning rate from the provided 5e -4 to 2.5e -4 , and increased the target network update frequency (referred to as C in [53] ) from 128 to 500, which is CleanRL's value. This improved both the agent's stability and average return.Our results in Figure 8 show that AD achieves strong performance across all 4 tasks, learning more stably and achieving higher average episodic return than the DQNs in all environments. Most notably, in comparison to DQN, AD more consistently reaches the maximum 500 return in Cart Pole, and achieves significantly higher return in Mountain Car. It also surpasses DQN's performance slightly in both Lunar Lander and Acrobot. These results suggest that AD is highly competitive against ANGC in these tasks. However, we note that as ANGC and AD learn with different signals, they are not competing approaches, and may possibly be integrated to achieve greater performance-we also look forward to exploring this potential in future work.We show the results of our main experiments with 95% bootstrap intervals in Figure 9 , following the recommendations by [55] to encourage better comparison of RL algorithms and reproduction of their results. For the MinAtar environments, we also show the ablation study results with 95% bootstrap intervals. Using the RLiable library provided by Agarwal et al. we aggregate the performance of AD and baselines on the Minatar and Mujoco tasks. Since the MinAtar and DMC environments do not offer human-normalized scores, we used the performance of common baseline algorithms (SAC and DQN, respectively) for normalization.We find that AD is within the confidence interval of TD-MPC2 on Mujoco and DQN on Minatar (compare Figure 10 ). It shows better mean and median performance than SAC on the Mujoco tasks. IQM excludes the harder hopper environment, which results in slightly worse performance of AD on this metric, as our method shines in this task.A limitation of our AD cells is that the W att can become expensive to compute and to learn. One trick we employed in DMC environments to improve the computational efficiency of our AD cells is to learn the Watt matrix as a product of two smaller matrices, one |A| × k and the other k × d where k is a small constant. For small k, the number of parameters in the matrix product is significantly smaller than the original W[l] att matrix, especially when either |A| or d are large, allowing for more efficient learning at the cost of expressiveness. We found that empirically in the DMC environments using k = 8 does not notably negatively impact performance while reducing run times. To further investigate the generalizability of AD across different learning tasks, we also evaluate AD on two supervised learning datasets, MNIST and CIFAR-10, and present our results here. Q-learning is a regression problem; to adapt our architecture to solve these classification problems, we simply change the loss function to cross entropy, and make each of the tanh weight layers correspond to a class, instead of an action. Using a 4-layer AD network with output sizes 2000, 2000, 2000 and 2000, we achieved 98.74% test accuracy on MNIST, and 57.25% test accuracy on CIFAR-10. These results are in line with the results achieved by Forward Forward in [30] .We show our results with different sized networks in Table 1 . As each layer in the network makes its own prediction, we additionally show the test accuracy of every layer. Remarkably, for almost every network size, the test accuracy steadily increases per layer. Like our experiments in RL, this supports that AD cells can learn to coordinate with each other for better performance without the need to backpropagate error signals across layers.For clarity, we want to note that these experiments are performed simply to add another perspective to evaluate the robustness of AD's learning across tasks. AD is intentionally designed for Q-learning, rather than supervised learning, as it imitates biological processes of reward-based learning.We ran our experiments on a shared scientific computing cluster using an assortment of CPU and GPUs. Each instance is run with a GPU (to improve the speed of training the neural network) and 12 CPU cores (to improve the speed of the environment simulation), and 20GB of RAM. On an Nvidia RTX 2080 GPU, a full training run of AD takes approximately 5 hours on the MinAtar environments, and 3 hours on the DMC environments. On an Nvidia A100 GPU, the run takes approximately 3.5 hours on MinAtar, and 2.5 hours on DMC. We additionally expended compute resources for hyperparameter tuning and ablation studies. In total, our experiments consumed approximately 9000 hours of compute using the aforementioned instances.While we did not prioritize computational efficiency in the current design of our algorithm, we note that like Chen et al. [15] , Lillicrap et al. [38] , Journé et al. [34] and other biologically-plausible deep learning algorithms that do not suffer from the weight transport and update locking problems, our algorithm can be adapted for neuromorphic hardware that keep data local to computational units [33] . These hardware have the potential for significant energy efficiency gains.We utilize several existing assets in both the implementation of our algorithm and our experiments.In Table 2 we provide their versioning, licensing and and URL for easier reproduction of our results.Version License URL Jax [12] 0. Our work introduces a new learning algorithm that computationally demonstrates how the credit assignment problem may be solved in the nucleus accumbens, where reward prediction errors are synchronously distributed. This research direction may yield significant positive impacts to healthcare and the social sciences. By improving our understanding of biological reward-based learning, our research has the potential to inform novel therapeutic strategies for neurological disorders characterized by dysfunctional reward processing mechanisms. By better aligning AI with biological intelligence, we also provide social science with more faithful AI agents for simulating and studying human social behavior, a practice that is increasingly adopted.However, aligning neuroscience and AI is not without risk of negative social impacts. The development of more faithful AI agents may encourage social sciences to increasingly replace human subjects, as AI agents may be cheaper or more malleable to certain experimental setups, and exogenous variables may be more easily controlled for. Yet AI agents are inherently biased by the data they are trained upon; replacement of human subjects may further marginalize underrepresented communities, which is already a large challenge in AI fairness research.In addition, deep reinforcement learning research, such as our work, consumes considerable amounts of computational power and energy (see Appendix K). While such research often yields valuable insights into intelligence, these gains should always be balanced against the carbon emissions caused by large-scale experiments.In Seaquest, where the performance difference between 3-layer AD, AD without forward-in-time connections, and single layer AD is most apparent, the latter agents' struggle may be attributed to the failure of learning the resurfacing mechanism, which traps it at a local minima. In this environment, the agent must regularly resurface while carrying a diver to replenish oxygen. In the short term, the agent can acquire more reward if it prioritized attacking enemies, rather than managing oxygen, but if it runs out of oxygen the episode ends. We observed that the AD agent without forward connections and the single layer agent both struggled to manage oxygen, often resurfacing randomly or without first acquiring a diver. In contrast, the AD agent with the forward connections learns to manage their oxygen more optimally, and often sacrifices short-term rewards to maintain higher oxygen. In fact, during some trials the AD agent became ultraconservative with oxygen, which led to the agent surviving on average over 1500 timesteps per episode over several thousand episodes before it changed strategy. Interestingly, this same behavior was not observed in DQN agents, whose average episode lengths are consistently under 800. This suggests that AD has the potential to learn completely different policies from DQNs.Biologically inspired learning algorithms seek to reconnect machine learning with neurobiology. This supports better transdisciplinary collaboration across ML and neuroscience, which may be essential to advancements in both biological and artificial intelligence research [71] .Similar to our work, Ororbia and Mali [53] proposes a deep Q-learning algorithm, ANGC, based on the Neural Generative Coding framework [51] , which is centered around the idea of active inference [54] . We indirectly compare the performance of AD and ANGC in Appendix C. Also closely relevant, Guerguiev et al. [25] proposes a BP alternative that solves credit assignment for supervised learning tasks, and show promising results on the MNIST dataset [22] . They point out that current neuroscience lacks sufficient understanding of how credit assignment is performed in the brain, and provide a method to train neural networks using segregated dendrites.Many other related works focus on other aspects of biological plausibility in deep learning; we mention a few here for the interested reader. Lillicrap et al. [39] , Lillicrap et al. [38] , Kunin et al. [36] , Akrout et al. [3] and Lee et al. [37] proposed algorithms that tackle the weight transport problem and non-locality of errors, which have long plagued backpropagation [24] . Journé et al. [34] adopts a Hebbian learning approach to address weight transport, locality and update locking. Miconi et al. [43] improves the plasticity of neural networks, inspired by the brain's mechanisms of neuromodulation. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: Our work does not involve human subjects.NAc dopamine also serves many roles beyond signaling reward prediction errors[18]; its full responsibilities are an active area of research. We only focus on its role in error signaling, which is most pertinent to our problem.For clarity, this is different from the temporal credit assignment problem, oft discussed in RL literature.38th Conference on Neural Information Processing Systems (NeurIPS 2024)."Forward" and "backward" are widely used in deep learning literature both to describe direction in time and in the order of layers. This can be confusing. For the remainder of this paper, we use "forward/backward" when describing time, and "upper/lower" when describing position among layers, as shown in Figure2.We give a simplified overview of reward-based learning in the NAc and its connections to TD learning here to provide intuition; this is not a complete picture of biological reward-based learning. There exists competing theories for alternative mappings of the TD learning framework to the brain's reward circuit; see Section 6.Note that AD does not assume action selection need happen in the NAc. Rather, we just take the assumption that the NAc computes the Q-value function, which takes in the state and action as parameters, and performs Q-learning. The action may be provided from another region like the dorsal striatum. 
 
 Appendix:  


Decision: There is broad agreement amongst the reviewers that the biologically-inspired algorithmic contribution of the paper is interesting, novel. and significant, both because it demonstrates an RL method without backpropagation and because it builds upon constraints from hypotheses in neuroscience. The main points of contention concern the extent to which the paper presents unsettled neuroscientific hypotheses as fact and the extent to which the paper offers or claims to offer insight regarding biological systems. Especially after the discussion period, most of the reviewers agree that the paper is sufficiently clear about its intent and its claims though there is some disagreement on this point.

The overall assessment seems to be that the paper is interesting on a technical/conceptual level but that the paper could be stronger in its communication with the neuroscience community. I recommend acceptance, but I also urge the authors to carefully consider the critiques from these reviewers and, while preparing the camera-ready version, to focus on clear, up-front statements about the disputed nature of the underlying neuroscience hypotheses and about the limited nature of any biological insights.

Reviews: review: paper_summary: paper_summary: An novel, biologically plausible RL algorithm based on DQN and Fast-Forward called Artificial Dopamine is introduce. The algorithm is inspired by Dopamine pathways in the human brain and uses synchronous, locally computed per-layer TD errors for training. Additionally, a novel recurrent network architecture is proposed where layer outputs are used as inputs to the respective preceding layer at the next time step. The approach is evaluated on a range of discrete and continuous RL tasks.
strength_weakness: strength_weakness: Minor Issues:
- L 116: "and is" -> "and are"
- L134: "in many practical applications, such as TD learning in the brain". Makes it sound like this a proven fact that there is TD learning taking place in the brain...
questions: questions: - L252: "Since the local updates are distributed and per-layer, they can be computed in parallel." The temporal dependency between time-steps due to the recurrence is still present. So, does this actually translate to improved runtimes?
limitations: limitations: The Limitations such as the use of the use of a layered structure, and limited proof for the method implemented (Q-learning) occurring in the brain are discussed in detail.
paper_summary: paper_summary: In this work, the Authors aimed to propose a computational model that is consistent with classical functions assigned to select regions in the reward system in the brain. To this end, they build upon George Hinton’s Forward-Forward algorithm, extending it with the ability to robustly generate continuous (instead of categorical) values and endowing it with the DQN loss. The Authors show that the proposed algorithm reaches competitive scores on a panel of standard RL benchmarks, and discuss its potential mapping on the brain circuitry.
strength_weakness: strength_weakness: The paper’s neuro premise (Sections 1; 2.1) reads as if the roles (or even activities) of the brain regions involved in the brain circuit are known undebatable. Even though this impression is then refuted in Section 6 (which is commendable), most of the paper operates under this assumption. While the paper correctly cites some classical literature on the topic, the debates over the brain region roles proposed in these works led to more nuanced models, e.g. the mappings of the actor-critic model on the reward circuit, and then to an overall notion that such mappings represent a bird-eye view of the problem and may not necessarily be mechanistically accurate. It would be nice to explore and discuss this uncertainty, as it may alter the ground truth for this work.

The paper’s comp neuro premise limits itself to the Forward-Forward algorithm. Although some of the other related work is mentioned in Section 7, these and other works have not made it into the model’s design. Meanwhile, the field of biologically plausible learning, including RL, is immense. There’s a host of work on predictive coding and local learning rules; the new works are submitted to (almost) every major ML conference. There are noteworthy works on the computational role of dopamine and RL in the brain, including the works on "Backpropamine" and DeepMind’s modeling of distributional dopamine coding. In that sense, it’s unclear why the current submission focuses on a single model, mostly validated within the ML community while ignoring the host of models validated within the computational neuroscience community. This may be mitigated by the Authors’ comment that their goal was to present a model consistent with neuroanatomy, which is valid, but brings me to the next point.

A single model consistent with data doesn’t preclude the existence of multiple other models, potentially with different mechanics, leaving the chance that the proposed model has little to do with the biological reality. Indeed, a Monte Carlo – or even brute force model – will eventually learn an approximately correct Q-function and won’t need backprop / coordinated feedback signal to do so. While – and I would like to stress that – the results in this paper are correct and the wording is correct as well, arguably, to learn something from this result, one would need to perform multiple additional comparisons between the (versatile, detailed) neuroanatomical structure of the reward circuit and a panel of prominent computational models. As such comparison is not performed within the scope of this work, it is arguably too early to recommend this paper for publication in NeurIPS at this point.
questions: questions: N/A
limitations: limitations: Related literature on neuroanatomy and computational models of the reward system are not considered.

The proposed model is not contrasted against other candidate models and thus may be uninformative.
paper_summary: paper_summary: This paper addresses the distributed credit assignment problem in biological reinforcement learning, where a naive implementation of backpropagation is implausible. The authors show that it is possible to update action values using a variant of the forwward-forward algorithm specialized for RL. They then validate that this algorithm is effective on a range of standard benchmarks.
strength_weakness: strength_weakness: - p. 5: The review by Glimcher (2011) is cited to support the claim that NAc (ventral striatum) neurons signal action value. In fact, there is a long tradition arguing that it is actually the dorsal striatal subregions (caudate and putamen) which signal action values, whereas ventral striatum signals state value (i.e., a form of actor-critic architecture; see Joel et al., 2002; O'Doherty et al., 2004). Most studies I'm aware of have shown that signals related to action value are relatively weak or absent in ventral striatum (see for example Ito & Doya, 2015).

- Overall I felt that the biological evidence for the modeling framework was weak. What is the evidence that dopamine updates values locally in the NAc? This is physiologically tricky because dopamine acts postsynaptically through volume transmission, diffusing broadly in the extracellular space. What is the evidence for the particular form of hierarchical message passing architecture proposed here? Basically, I felt that the ideas here are interesting in principle but as presented are rather disconnected from empirical data.

Minor:

- p. 2: "connection ," -> "connection,"
questions: questions: - I was unsure whether the claim here is that the NAc is a multi-layer network or one layer in a multi-layer network.

- If the authors would like their model to be taken seriously by neurobiologists, they need to do more to directly link it to empirical evidence.

- Ideally the authors could make some neural predictions based on the model. It would also be helpful to understand how the predictions are different from those of other models.

Rebuttal update: I have raised my score from 6 to 7 following the authors' responses to my comments.
limitations: limitations: The authors extensively discuss limitations in section 6. There are no potential negative societal impacts.
paper_summary: paper_summary: In the paper, the authors propose a novel reinforcement learning algorithm. The algorithm is based on Q-learning and uses a  biologically inspired design to work with local error signals and updates, eliminating the need for the biological implausible backpropagation. The proposed algorithm, Artificial Dopamine, is motivated by the dopamine based learning system within the brain, concretely the ventral tegmental area (VTA) and the nucleus accumbens (NAc). It is realized with a novel recurrent layer/cell type, that predicates Q values from local error signals with an attention-like mechanism. The algorithm is evaluated against several baselines in a set of established RL benchmarks, and additional ablations add further insights on the algorithm.

---
**Post-Rebuttal/Discussion**: During discussion with the authors and fellow reviewers, open questions were tackled, and additional results were presented. I've increased my score to 8 due to this.
strength_weakness: strength_weakness: While the majority of the paper is well written and understandable, one particular details is unclear and different sections seem to be contradicting. The AD cells/layers use multiple tanh layers, one for each action as stated and visualized. In Section 3.1, however, there is only one weight matrix W_att mentioned and used. Also in Figure 3 there is only the Q_t value, independent of an action. Inside each AD cell, are there multiple layers, one for each action, and if so, how are they used together for Q? Or is it one layer/matrix, with one dimension per action? (The first seems more plausible, but clarification and unification of notation and description would be beneficial.)

While computational demanding and, hence, understandable, 10 runs are quite low for comparing RL algorithms (known problem in the community, although often neglected/ignored). Recent frameworks and metrics have been proposed tackled at these problems related to low number of runs and comparing overall algorithms performance [1]. It would be beneficial to add such metrics (e.g., performance profiles and IQM/IQR) to the paper for a better comparison of the proposed algorithm.

------------

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021)
questions: questions: The algorithm is based on Q-learning, could the main ideas be transferred to other RL backbones?

The reward needs to be globally available to all layers to compute the local prediction errors. Which implications does this global target induce? Any theoretical or practical drawbacks?

How crucial is the choice of the two activation functions in the AD cell?

Typo (?) Line 101 mentions 15 tasks, while the rest of the paper mentions 14. Additionally, the main paper actually ‘only’ uses 10 tasks, the additional 4 are only provided in the Appendix for comparison with an additional baseline. Can this be clarified or better, unified in the main paper?
limitations: limitations: Limitations are discussed thoroughly in a separate Section. While detailed, they focus solely on the biological background, not mentioning technical/practical limitations.
Some of such limitations, like only working with discrete/discretized action spaces and the difficulty in scaling to larger action spaces, are mentioned in other parts of the paper, it would be good to add such practical limitations and assumptions in the dedicated limitation section.

Comments: title: raising score
Thanks.
Discussion Period
thanks
comment: Reviewer dpw2 is correct: we don’t claim that AD is a mechanistically accurate model of real brain activities. Our paper focuses on an algorithmic insight relevant to biology—demonstrating that distributed error signals, as observed in the brain, can be sufficient for reward-based learning—rather than on biological modeling per se. We propose a more biologically plausible RL algorithm that eliminates the need of BP in order to do so.

We want to again express our gratitude towards the AC for diligently facilitating the discussions. Quoting Reviewer dpw2, we wish it would be like this on all papers.
After careful consideration, based on the Reviewers' reports and Authors' feedback, I raise my score to borderline to best reflect the current state of the work, under the NeurIPS guidelines.
Thank you for the additional effort in clarifying concerns and providing additional results. There are no open issues from my side and I still think the paper is a valuable contribution to the conference.

@AC: Also thanks to you for your engagement! I'd wish it would be like this on all papers...
Thank you, Reviewer dpw2, for your positive response and for taking the time to review our answers. We also appreciate Friendly Neighborhood AC for checking in on us!

We’re pleased to share that we have now generated the IQM/IQR plots (apologies for the delay—our resident RL expert is traveling): https://anonymous.4open.science/r/AD_neurips_rebuttal-3723/IQM.pdf

Regarding the DMC environments, the data for the SAC and TD-MPC2 baselines (taken from standard implementations by Denis Yarats & Ilya Kostrikov, 2020, and Hansen et al., 2023, respectively) had fewer seeds, which made individual environment comparisons less meaningful due to the low number of runs. Therefore, we opted to aggregate performance across the MinAtar tasks and the DMC tasks and present the aggregate statistics. Since the MinAtar and DMC environments do not offer human-normalized scores, we used the performance of common baseline algorithms (SAC and DQN, respectively) for normalization. Our plots display the IQM (with range), median, mean, and optimality gap.

Additionally, to demonstrate that our performance is consistent across each individual tasks, we computed 95% bootstrap confidence intervals for each task, following the method recommended by Patterson et al. (2023) for RL best practices. These are plotted here: https://anonymous.4open.science/r/AD_neurips_rebuttal-3723/boostrapped%20CI.pdf

We will ensure that the style and formatting of these plots align with the other figures in our paper, and we will include them in the camera-ready version if accepted. Please let us know if you have any further questions or concerns.

---

**References**

Denis, Y. and Kostrikov, I. (2020). Soft actor-critic (sac) implementation in pytorch. https: //github.com/denisyarats/pytorch_sac.

Hansen, N., Su, H., & Wang, X. (2023). Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828.

Patterson, A., Neumann, S., White, M., & White, A. (2023). Empirical design in reinforcement learning. arXiv preprint arXiv:2304.01315.
Dear Area Chair aQyN,

thanks for doing everything to clear the remaining questions toward the informed decision on the paper.

Your summary is correct.

I believe that the point I'm raising is not a matter of editing a few critical places in the text but rather of a general message. I welcome the changes introduced by the Authors regarding further disclaimers on the extent of the biological plausibility of the model, however, starting from the abstract, the text has a clear biological focus - rather than an algorithmic focus - so my concerns remain. Fortunately, we have multiple Reviewers here, so my take on the text only matters that much. I also acknowledge the Reviewer dpw2's comment on not perceiving it as a significant issue (along with other Reviewers' positive reports, indicating the same assessment of the text). In that light, I'll consider revising my score, though it will take longer.

To the specific points:
1. Resubmission. I've only mentioned it regarding the Authors' question on the previous score vs. the current score. The only point I'm making in that regard is that the previous conference score first reflected the initial uncertainty and then the time limitations of the rebuttal period while none of these are the factors here at NeurIPS, hence the different number.
2. While all the outlined changes are indeed welcome, they mostly address the algorithmic part of the work, thus being tangential to my concerns. Hypothesis falsifiability sounds interesting though and I'll be happy to read the related parts of the text shortly, open to revisiting my score.
Hi reviewer Sdew,

We are about to lose access to the authors (August 13, 11:59pm AoE). Let's try to clear up as many points of contention as we can before then.

From your comment it sounds like your primary concern is that the paper may still give the impression that it is presenting a model of biological processes rather than, per the authors stated intent, a biologically inspired algorithm that demonstrates the feasibility of a hypothesized biological phenomenon. Is this accurate or are there other critical issues that have not been addressed by the additions to the paper (summarized succinctly in the authors' most recent comment)?

Vis a vis that concern, I see that reviewer dpw2 has commented that they didn't perceive this to be a significant issue in the paper and that they felt clear about the intent and the contribution. So there is at least some evidence that some readers are not misled. Are there critical places in the paper that you feel are particularly misleading or are in particular need of clarification of this point? Perhaps in the brief time we have left, the authors could offer some alternative wording that might sharpen the intent of the paper. Do you believe that addressing this issue this would be a matter of a few well-placed, clearly stated sentences or significant revision of the entire work?

Finally two quick points:

1) This paper is not in a resubmission track and extent of change from a submission to a different conference is not part of the NeurIPS acceptance criteria. It may certainly be that weaknesses of that submission have carried over, but let's please set aside "scope of the changes" in the assessment and focus on the merits of the manuscript that was submitted to NeurIPS.
2) Considering the authors' summary of the additions to the paper, please take a look back at your review and consider whether any of the comments based on the old manuscript are no longer applicable; if so, please consider revising the review to reflect the current manuscript.

Thanks!
Hi reviewer dpw2 and authors,

dpw2: Just to clarify, when you say "Also waiting for...," do you think these additional figures are critical for the paper's completeness? That is, would your overall assessment go down if the authors did not offer these additional figures?

Authors: Are you able to provide the IQM and IQR from your experiments?
Thank you for engaging. We have made several substantive improvements in the paper since our last submission that we’ll highlight below, and we’ll edit the paper in revision to make these changes more clear. Some additional figures were in the new appendices, but if important can be moved into the main text.

1. **Algorithm Extension**: We designed and evaluated an extension of our algorithm to learn distributions of values, aligning our work with Dabney et al. (2020) (Section 5 & Appendix M).
2. **Sample Efficiency**: We significantly improved the sample efficiency of our algorithm in the DMC environments (Section 5).
3. **Ablation Studies**: We conducted additional ablation studies on DMC and MinAtar environments, analyzing the effects of varying layer sizes in a single-layer AD network (Section 5 & Appendix L).
4. **Biological Assumptions**: We expanded our discussion on biological assumptions and alternative hypotheses, citing eight additional neuroscience works (Section 6), and added a forward reference to address your concern about potentially misleading readers.
5. **Hypothesis Falsifiability**: We defined the criteria of sufficiency to explicitly make our hypothesis falsifiable (Section 4).

We agree that our work is more algorithm-focused, and we do not claim that our model is a realistic biological reconstruction of specific brain mechanisms. However, our motivation for developing this algorithm is rooted in biological observations, and we believe our contribution holds relevance for the field. Specifically, our paper focuses on an algorithmic insight relevant to biology—demonstrating that distributed error signals, as observed in the brain, can be sufficient for reward-based learning—rather than on biological modeling per se.
Yes, I will raise my score from 6 to 7.
Thanks for acknowledging our rebuttal. We appreciate your feedback and are committed to reflecting your suggestions in the revised paper.

To address your concerns, we plan to make the following updates:

1. Clarify that AD does not require action selection to occur in the NAc and acknowledge competing theories (Ito & Doya, 2015; Weglage et al., 2021) in the Limitations section.
2. In the background section, clarify that dopamine providing local signals for updates is supported by the findings of Wightman et al. (2007).
3. Add to the Limitations section that our hierarchical message passing architecture is a design choice from deep learning practices and not a fully mechanistic model of the NAc.

We'd value your feedback on whether these updates address your concerns and if there are any other critical points we should prioritize to potentially raise the score.

Thank you again for your constructive feedback.
Thanks for the follow-up question. You’re right— we can save the last timestep’s activations along with the sequences into the replay buffer, and use these values for training, but there would be a bias in the gradients as the model parameters are updated. Instead, to avoid this, we store short sequences during rollout into the memory buffer, then replay the sequences during training, zero-initializing the first "recurrent" state. This follows established practice in deep RL for training RNNs (Hausknecht & Stone, 2015; Kapturowski et al., 2018). Does that answer your question?

---

**References**

Hausknecht, M., & Stone, P. (2015, September). Deep recurrent q-learning for partially observable mdps. In *2015 aaai fall symposium series*.

Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., & Dabney, W. (2018, September). Recurrent experience replay in distributed reinforcement learning. In *International conference on learning representations*.
Thanks for this thread.

In general I agree to the high-level summary, although the critique on *"the extent of the biological evidence that this model could correspond to real activity/mechanisms in brains"* is less an issue in my opinion. That was not the intend of the paper nor did the authors claim this (please correct me if I'm wrong), but rather aimed to show a *bit more biological plausible* RL algorithm that is *inspired* by neuroscience and recent ML results (ff), and that eliminates the need of BP.
Thank you for your detailed answer to my review and questions. The answers clarified the open questions and I hope to see the mentioned text changes in a potential final paper.
Also waiting for "We’re working on additional figures that show the IQM and IQR of our data for our main results and ablation study, and will upload these before the end of the discussion period." for a better comparison of the methods.
Thank you for starting the discussion. I see your summary as correct.
I would like to thank the Authors for their detailed response. Below, I address the Authors' specific concerns and elaborate on my report.

Regarding the similarity to the previous review, I would like to describe the way I've prepared this one to clear all possible concerns in this aspect. First, I've compared the new version of the paper with the old one. To do that, I went through the main text of each paper, comparing them paragraph by paragraph. What I found is that, besides the minor formatting changes aimed at accommodating the NeurIPS format, there were three paragraphs added: the first paragraph in Section 4, the last paragraph in Section 5, and the middle paragraph in Section 7. Second, I went through my old review to see the points that I still view as valid, considering the added paragraphs and the extended discussion with the Authors last time, which has provided me with a lot of important details regarding this work (the Authors' engagement was much appreciated). The added experiments did not contrast the proposed model with other existing models, and the disclaimers on the biological plausibility, while more abundant now, were still sparse. Finally, to determine my score, I've factored in my previous assessment of this work and the scope of the changes made between the two versions. Specifically, while the previous rebuttal period was short, limiting the scope of changes that could be introduced during it, there's much more time between the submissions, so typically substantial revisions of the papers are expected.

Now to the score part. This was (and is) one of the harder submissions I've got to review (and score) for the following reasons. On one hand, the paper is technically correct. All appropriate disclaimers are made regarding the biological relevance of the work. On the other hand, the biological relevance is constantly alluded to throughout the text, potentially leading a reader to the idea that the paper explains how the brain works, and likely leading the reader to the conclusion that the paper has a high impact on the field of biology. Notably, most of the disclaimers and additional considerations are provided at the end of the main text and in the Appendix, way after the readers' opinions on the work are formed. This concern is consistent with the reports of all other Reviewers here who have expressed concerns regarding the biological relevance of this work and/or the language describing such.

What I would see as a proper way to present this work in its current scope is to be more upfront about the limitations, such that the reader would immediately know that the mechanistic modeling of the brain was not a goal or a result of this work, but rather a new computational architecture was tested, inspired by some neuroscience works. Then, the paper could be assessed based on its direct contribution rather than on what some may see as an alluded promise. The algorithm itself is a valid contribution and, as such, should absolutely be published somewhere.

Not that it matters, but I would be happy to revise my score to borderline contingent on the changes in the message from biology-focused to algorithm-focused. I'm happy to re-read the updated anonymous PDF with such changes. Please also let me know if there are other changes or considerations that I may have overlooked.
Thank you for the given answer. However, it is still unclear to me how the model could be parallelized.

Assuming you are training weights that process the last timestep's input, you will need to know the input before updating those. I guess you could buffer the intermediate layer's activations during rollout for later training, entailing a bias in the gradients when you update the model more than once on the same batch of collected data?
Hi all! Just your friendly area chair checking in.

First, thanks to the reviewers for your work so far on this paper. The discussion period has begun, which includes both reviewers and authors. This is our opportunity to clear up misunderstandings, get answers to questions, and generally gather enough information to make an informed decision grounded in the acceptance criteria.

It seems there is general agreement that problem addressed by the paper (developing biologically plausible RL algorithms at scale) is important and that the paper is well-presented. The main critiques raised seem to concern insufficient acknowledgement of alternative neuroscientific hypotheses and biological models, and the extent of the biological evidence that this model could correspond to real activitiy/mechanisms in brains. Does that seem like a fair high-level summary?

Reviewers: please carefully read the other reviews and the authors' responses (with included PDF) and let us know what follow-up questions you have and to what extent your evaluation might change as a result of the additional context. Please especially raise any points of disagreement with other reviewers or the authors, as these are opportunities for us to clarify misunderstandings and detect misconceptions.
Thanks for your thorough response to my comments. I hope that at least some of this will be reflected in the paper.
**References**

Roesch, M. R., Singh, T., Brown, P. L., Mullins, S. E., & Schoenbaum, G. (2009). Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. *Journal of Neuroscience*, *29*(42), 13365-13376.

Mannella, F., Gurney, K., & Baldassarre, G. (2013). The nucleus accumbens as a nexus between values and goals in goal-directed behavior: a review and a new hypothesis. *Frontiers in behavioral neuroscience*, *7*, 135.

Ashton, S. E., Sharalla, P., Kang, N., Brockett, A. T., McCarthy, M. M., & Roesch, M. R. (2024). Distinct Action Signals by Subregions in the Nucleus Accumbens during STOP–Change Performance. *Journal of Neuroscience*, *44*(29).

Weglage, M., Wärnberg, E., Lazaridis, I., Calvigioni, D., Tzortzi, O., & Meletis, K. (2021). Complete representation of action space and value in all dorsal striatal pathways. *Cell reports*, *36*(4).

Wightman, R. M., Heien, M. L., Wassum, K. M., Sombers, L. A., Aragona, B. J., Khan, A. S., ... & Carelli, R. M. (2007). Dopamine release is heterogeneous within microenvironments of the rat nucleus accumbens. *European Journal of Neuroscience*, *26*(7), 2046-2054.