Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
Authors: Zequan Wu
Mengye Ren
Venue: ICLR 2026 Poster (2026)
Keywords: neural network
localized learning
reinforcement leaning
biologically plausible learning
Primary area: reinforcement learning
TL;DR: We propose ARQ, a backprop-free RL method using layerwise goodness and action conditioning, achieving strong performance on MinAtar and DeepMind Control Suite.

Abstract: The Forward-Forward (FF) Algorithm is a recently proposed learning procedure for neural networks that employs two forward passes instead of the traditional forward and backward passes used in backpropagation. However, FF remains largely confined to supervised settings, leaving a gap at domains where learning signals can be yielded more naturally such as RL. In this work, inspired by FF's goodness function using layer activity statistics, we introduce Action-conditioned Root mean squared Q-Functions (ARQ), a novel value estimation method that applies a goodness function and action conditioning for local RL using temporal difference learning. Despite its simplicity and biological grounding, our approach achieves superior performance compared to state-of-the-art local backprop-free RL methods in the MinAtar and the DeepMind Control Suite benchmarks, while also outperforming algorithms trained with backpropagation on most tasks.

Published as a conference paper at ICLR 2026
LOCAL REINFORCEMENT LEARNING WITH ACTION- CONDITIONED RoOT MEAN SQUARED Q-FUNCTIONS
Frank Wu
Mengye Ren
Carnegie Mellon University Pittsburgh, PA, USA frankwu2 cmu. edu
New York University New York, NY, USA
mengye@nyu.ed edu
ABSTRACT
The Forward-Forward (FF) Algorithm is recently proposed learning procedure for neural networks that employs two forward passes instead of the traditional forward and backward passes used in backpropagation. However, FF remains largely confined to supervised settings, leaving a gap at domains where learning signals can be yielded more naturally such as RL. In this work, inspired by FF's goodness function using layer activity statistics, we introduce Action-conditionnd Root mean squared Q-Functions (ARQ), a novel value estimation method that applies a goodness function and action conditioning for local RL using temporal difference learning. Despite its simplicity and biological grounding, our approach achieves superior performance compared to state-of of-the--rt local backprop- free RL methods in the MinAtar and the DeepMind Control Suite benchmarks, while also outperforming algorithms trained with backpropagation on most tasks.
INTRODUCTION
The success of deep learning has relied on backpropagation (Rumelhart et al. 1986), a procedure that has significant limitations in terms of biological plausibility as it requires synchronous com- putations and weight symmetry. Many works have provided backprop-free alternatives for training deep neural networks (Lillicrap et al., 2016 Nøkland [016 Nokland & Eidnes (019). Notably, Hinton (022) proposed the Forward-Forware algorithm (FF), new approach that performs lay- erwise contrastive learning between positive and negative samples, This algorithm lightweight and entirely eliminates the need for backpropagation, thereby addressing some of the biological plausibility concerns.
However, most studies on backprop-free methods are focused the search for a biologically plau- sible mechanism for performing gradient updates on supervised tasks. Could a biologically plausi- ble source of learning signals be equally meaningful? Reward-centric environments and temporal- difference (TD) methods (Sutton] 1988) serve as natural candidates for filling this gap. Biological brains have evolved through series of reward-guided evolution. while ample evidence has shown that our brains could be implementing TD (Schultz et al. 1997a, O D'ohertty et al. 2003 Watabe- Uchida et al. 2017 Amo al. 2022). Since the goodness score FF models the "compatibility" between the inputs and labels, this local learning paradigm can be readily adapted to a reinforcement learning (RL) setting where we model the value of an input state and an action from each layer's activities. See Figure [ for comparison between the supervised learning and RL setups of the forward-forward learning paradigm.
Towards integrating local methods and Ru., et al. (2024) recently proposed Artificial Dopamine (AD) that incorporates top-down and temporal connections in an Q-learning framework. Since the local Q-Function estimation needs to be explicitly predicted, Guan et al. (2024) uses dot-product between two sets of mappings from the inputs that produces the value estimate for each action. This design, while backprop-free, makes the architecture more flexible modeling complex inputs. However, AD still relies on the output of the dot-product to be the same dimension as the action space, limiting the performance of the method.
Inspired by FF's local goodness function from using layer statistics, we propose Action-comditioned Root mean squared Q-Function (ARQ), a simple vector-based alternative traditional scalar-based

---

Published as a conference paper at ICLR 2026
"5" y
"Left"
G2
Q1
Q2
QL
Local Supervised Learning
b) Action-Conditioned
Local Reinforcement Learning
Figure 1: Local learning paradigmms inspired by the Forward-Fowwar (FF) algorithm (Hinton 2022). The original FF is designed for supervised learning, where each layer models the "goodness" between image & and label y. Information is carried forward only through bottom-up and optionally top-down connections without backpropagation. learning-ese each layer takes a state observation x and an action candidate a as input, and estimates the value by taking the root mean squared function of the hidden vector.
We extend FF local learning for reinforcement
Q-value predictors designed for local RL. ARQ is composed of two key ingredients: a goodness function that extracts value predictions from vector of arbitrary size, and action conditioning by inserting an action candidate at the model input. ARQ significantly improves the expressivity of local cell by allowing more neurons at the output layer without sacrificing the backprop-free prop- erty. By applying action conditioning, we further unleash the capacity of the network to produce representation specific each state-action pair. Moreover, ARQ can be readily implemented on AD and take full advantage of their non-linearity and attention-like mechanisms.
We evaluate our method on the MinAtar benchmark (Young Tian 2019) and the DeepMind Con- trol Suite, challenging suites designed to test RL algorithms in low www-dimensional settings where local methods remain viable. Our results show that our method consistently outperforms current local RI methods and surpasses conventional backprop-based value-learning methods in most games, demonstrating strong decision- makinn capabilities without relying on backpropagation, Through this contribution, we seek to encourage further exploration of the intersection between RI. and bio- logically plausible learning methods.
RELATED WORKS
Backprop-free learning methods & FF: In recent years, several backprop-free training algorithms have been proposed to address the limitations of traditional backpropagation in neural networks
(Lillicrap et al. 2016 Nokland 2016;; Nøkland & Eidnes 2019; BBliiovssy et al. 2019 Baydin
ct al. 2022: Ren al. 2023] Fournier al.J (ing) et al. 2023 Innocenti et al., 2025). One
notable method is the Forward- Forward-Forward Algorithm (FF) (Hinton 2022). which offers biologically
plausible, energy-efficient alternative to backpropagation. To extend the capabilities of FF, Ororbial & Mali 2023) proposed the Predictive Borward-Forwan Algorithm, showing that a top-down gen- erative circuit can be trained jointly with FF., Tosato et al. (2023) found that models trained with FF objectives generate highly sparse representations. This pattern closely resembles the observations of neuronal ensembles in cortical sensory areas, suggesting FF may be a suitable candidate for mod- eling biological learning. Recently, [Sun al. (2025) proposed DeeperForward,, integrating residual connections (He et al., 2016), the mean goodness function, and channel- wise cross cross-entropy based objective function (Papachristodoulln al. 2024) into FF. DeeperForward yields 87% on CIFAR- 10 with 17-layer deep architecture.
Value Estimation in Deep Neural Networks: TD methods for value estimation have been partic- ularly useful in the recent decade as the rise of deep neural networks offers a powerful function approximator. Mnih et al. (2013) introduced DQN, where a deep neural network is applied to ap- proximate the Q-Function. They showed that this method significantly outperformed earlier meth- ods on the Atari 2600 games, initiating family methods built upon this architecture (Van Hasselt et al. [2066] Wang al. 2016 Dabney et al. 2018b: Hessel al. 2018 Fortunato al. 2017 Dab-
ney et al.,) 2018a| Hausknecht Stone 2015). In actor-critic architectures, it also common to use
deep neural network for value and advantage estimation (Schulman al. 2017 2015a b Lillierap
2

---

Published as a conference paper at ICLR 2026
et al. 2013: Mnih al. 2016; Haarnoja al. 2018b a Fujimoto et al. 2018 Gruslys al. 2017
Abdolmaleki et al. 2018; Kostrikov et al. 2020; Yarats et al. (021). For planning- based methods
using either Monte Carlo tree search MS) or a learned model, value estimation is also significant
in driving the planning process (Schrittwieser et al. 2020 Silver et al. 2016 2017 Hansen et al.)
2023 Saaks al.
2024: Ye al.) [021]] Hafner al. 2023 2020 2019a
Yet, few works have
investigated the capability of local learning on value estimation.
Action Conditioning of Value Estimators: An important design choice in value estimation is whether the network is conditioned on the action. Early neural value estimation methods Riedmiller (2005) incorporated action conditioning by incorporating both state and action as model inputs. With the advent of deep neural network approaches such as DQN, practices began to diverge. Purely value-based methods like DQN are typically only state-conditioned, with action- spriificceccficccc predictions produced at the output layer by indexing over action values. This design is computationally effi- cient and well-suited for discrete tasks with low-dimensional action spaces. In contrast, actor-critic methods developed for high-dimensional continuous control tasks Lillicrap et al. (0015) Haarnoja ct al. 8a) condition on both state and action at the input of their critic networks. Although this distinction is largely arbitrary in backpropagation- based architectures and can be adapted to the task, we show that action conditioning at model inputs is strictly preferable for local RL.
Local and Decentralized Reinforcement Learning: The concept of decentralized RL can be dated back to the dawn of RI. Klopf (1982) introduced the idea of the hedonistic neuron, which hypoth- esized that each of neurons may be guided by their independent rewards. Instead of being miniscule part of a large operating neural system, each neuron may be an RL agent itself. In mod- emn RL literature, the localized formulation of RL methods can be related to the multi-agent RL (MARL) setup, where multiple independent agents can be designed to cooperate well toward max-
imizing their joint rewards Tan 1993 Foerster al., 2017 Palmer al., 2017 Su et 2022]
Lauer & Riedmiller 2000; Jiang & Lu |023]
De Witt et al. (2200]S [Su & Lu 2022
SS & Lu Arslan
Yükkeel 2016] Jin al.,
2021). Conveniently, we can frame the problem of training RI. using lo-
cal objectives as a MARL problem where each agent represents different modules within network.
Recently, Seyde et al. (2022) has explored a similar approach for the continuous control problem, showing that using separate critics network for each fixed action after action discretization works surprisingly well. Guan et al. (2024) builds upon the FF architecture, showing that a network with nonlinear local operations, decentralized objectives, and top-down connections across the temporal dimension can exceed state-of-the methods trained end-to-end, We extend upon this literature of decentralized methods for value estimation.
BACKGROUND
Borward-Forward (FF): The FF Algorithm (Hinton 2022). as its name denotes, uses two forward passes instead of one forward pass and one backward pass used in backpropagation. The first for- ward pass carries the positive data, or real data, while the second pass carries the negative data, or fake data either manually defined synthetically generated by the network. The network is then trained by maximizing the goodness of each layer in the positive pass, while minimizing the good- ness of each layer in the negative pass. The definition of goodness based on a hidden vector z is as follows:
G. =
Σ 2
In layman's 's terms, this equation represents the sum of squares of all activations over L, a measure of the magnitude and orientation of the activation vector. By training its layers greedily, FF is biologically plausible and could serve as a model for our future discovery of the inner mechanisms of the human brain.
Value Estimation in Deep RL: Estimation of the value function is core to RL. In layman's 's terms, the value function measures the expected sum of future rewards after discounting given current state. A similar formulation can be constructed when we are interested in the goodness of a state- action pair, which is usually termed the Q-Function. Formally.
ax = Br
k Re4R+1
St 8, A,
(2)
3

---

Published as a conference paper at ICLR 2026
Artificial Dopamine (AD) (Guan et al., 2024)
ARQ (Ours)
Q1
Rg
an
BB)²
hp41.
Cell
hyct1
ht+1
Cell
h
hz-1,
h4-1
ag
Xt
Figure 2: High-level computation diagram between Guan et al. (2024) and ARQ. Key implemen- tations of ARQ highlighted in red. AD cells take activations (highlighted in blue, darker color means earlier layer) and the state observation as input and produces a vector of size na- each indi- cating the value prediction of an action candidate. Our ARQ takes activations, the state observation, and the action candidate as input, and produces a hidden vector of arbitrary size, before passing through a root mean squared function to yield scalar prediction.
A widely used class of methods for value estimation is temporal difference (TD) learning (Sutton, 1988), which bootstraps value estimates by blending immediate rewards with future predictions, allowing for online, incremental updates. This method paved the way for the development of many subsequent approaches, particularly Q-learning. Take a Q-Function ((s, a). To update the function given experience (Se, a, S(+1). Q-learning makes the following iterative update
Q(++2) (S, A,) (S,, + a(R, + max (S+41, (S,,
(3)
where y is a discounting factor, a a pre-determined learning rate, and a' represents any possible actions in the next step.
Recently, the rise of neural networks pushed q-learning to new heights, Mnih al. (2013) proposed DQN, approximating Q-values using deep neural network. Based on the Bellman equation, DQN constructs a mean squared error function as the objective, namely
Lg = RR + max Qo
Qo(S,, A.))
(4)
Mnih et al. (2013) tested their agents on the Atari 2600 environment, and show that a convolutional neural network trained this fashion is able to achieve near-human performance level from raw pixel inputs, a feat previously considered far-fetched.
Artificial Dopamine (AD): AD (Guan al. 2024) trains local RL agent using Q-learning- An AD network is consisted of multiple AD cells, each of which makes an independent estimation of QSS,, At). To yield scalar estimation, each AD cell adopts an attention-like mechanism to compute a weighted sum of its hidden activations using weights from a separate linear projection, effectively incorporating nonlinearity while maintaining backprop-free. Additionally, each AD cell takes inputs from the layer below, the layer above, and also the raw state observation, enabling skip connections, top-down connections, and information flow throughout the temporal dimension in an RL environment. Mathematically. AD cell at depth / conducts the following operations,
X concat (8t, 1111111100 (1
(5)
K ReL,UW,
(6)
( a.) Lanh(XTW') ot1, W att, X)b
(7)
where h represents the activation of the AD cell at time and depth I. While this attention -like mechanism brings exciting nonlinearity to single AD cell without the need for backpropagation, the scalar nature of Q(s,, a,) implies that the dimensionality of Watt must be limited by the size of the action space. We aim remove this constraint.
4

---

Published as a conference paper at ICLR 2026
4 ARQ: ACTION-CONDITIONE ROOT MEAN SQUARED Q-FUNCTION
In the context of FF, the goodness function measures the likelihood of the observation to come from the postive distribution. In the context of RL, the concept of value measures the expected sum of future rewards for the trajectories starting from given state. We observe connection both denote a measure of the current input's 's desirability to an agent. Could the association between goodness and value be exploited to unleash the capacity of local RL networks? In this section, we introduce a novel vector-based training mechanism for local value estimation that can be used out-of-the- off-the-box. box. We term it the Action-conditioned Root mean squared Q-Function (ARQ).
4.1 ARQ
Take a state s and an action a. Based on the Bellman equation, we are interested in finding
(+ a) RE, Rs + 7 max (S++,, d' d'){s, = 8, A. =
(8)
a'
Inspired by the association between the concept of goodness from FF and the concept of value in RL, we directly approximate Q(s, a) using the goodness function. Given hidden vector 2, which can be either an intermediate action or an output embedding from a neural network. Instead of taking the sum of each vector unit squared, we make a small modification and take the root mean squared (RMS) function of the vector after mean subtraction to prevent its goodness values from exploding as we scale up the number of units. This is equivalent to the standard deviation of the hidden vector. In mathematical terms, we compute the estimated value of applying action a on state using
By E yi. Qo(s,a)= wey
IE Ey
(u/ - m,,)
V
where 0 denotes the parameters of the network and denotes hidden vector produced by the network.
To train this network, we update our weights using the same mean squared objective function as previous Q-learning methods (Mnih et al. 2013). Namely,
Lg = R₁ + ? max Q0( Qo))
(10)
Note that it is possible to sample positive and negative data in order to train in the same contrastive fashion as the original FF algorithm, particularly when method is used with training mechanism that maintains a replay buffer. We leave this for future investigations to keep our method versatile.
ARQ can be implemented out-of-the-box in place of the standard Q-learning formulation. Given any intermediate vector produced by an arbitrary neural network architecture, ARQ can extract scalar statistics that serve prediction for the estimated value without any parameters. This property allows architectures designed for local RL to enjoy greater flexibility.
Due to the nature of goodness functions to produce scalar values, it is natural to implement ARQ with action conditioning at the model input. Concretely, to estimate Qe(s, a), the neural network 0 takes both the state vector s and the action vector a as inputs and outputs single scalar prediction. This contrasts with implementations such as Mnih et al. (2013) and Guan al.] (2024), where the model receives only the state vector s and produces an output of dimension n. with each entry corresponding to the value of discrete action. We demonstrate in Section that this minor design decision is critical to the performance of local RL methods. For tasks with discrete action spaces, we use a one-hot vector to represent an action candidate. For tasks with continuous action spaces, we apply bang-bang discretization on the action space following Seyde et al. (2021) and condition the network on the binary action vector.
Action Conditioning:
4.2 IMPLEMENTATION
To evaluate our method against state-of-the- local RL architectures, we implement AR on top of Guan et al. (2024).
Our implementation is consisted of multiple cells stacked together, each of which takes inputs from the layer below, the layer above, the input observation, and an action candidate a, to make an estimation of (( a.). Each cell adopts similar attention-like like mechanism Guan et al. (2024). After the attention mechanism, we apply the goodness function on the intermediate vector after the attention computation. Specifically, cell at depth conducts the following operations,
5

---

Published as a conference paper at ICLR 2026
_/ X concat(st, M/ M 1 0e),
(11)
h X),,,
(12)
V/ = (anh(X""3s att, Watt, Ki
(13)
My = E Vi, ((s;,
(3) - myy) u.
E
(14)
Gradients are passed only within each cell to ensure the architecture is backprop-free.
Pseudocode for AD and ARQ is provided in Figure In both architectures, the intermediate quan- tities Z, Zg. and hy play roles analogous to the query, key, and value vectors in self-attention. The computation of 7 Z1 (Line 8, Algorithm 2) produces dimension-wise interaction map that de- termines how information is redistributed across latent dimensions, similar in spirit to an attention mechanism but applied over feature dimensions rather than token positions. The key distinction in ARQ is that Z2 is not restricted to have width nai instead, its dimensionality can be chosen freely. This flexibility allows ARQ to learn richer state-action interactions than AD, whose dimensionality is constrained by the cardinality of the action space.
Algorithm AD (Guan et al. 2024)
Algorithm 2 ARQ (Ours)
1: + Se. 1 11
1: X + [st. h4-1, =
2: 1/ + LayerNorm ReLU(W,
2: h/ +
ReW D Diieension
3:
Dimension: d
4: Z1 <- Watt, X
> Dimension: na X d
4: Repeat X along batch dim n. times
5: Z2 +- Watta X
D Dimension: dast X na
5: X f- [X, at)
D Action conditioning
6: W & Zr Zi
Dimension: xd
6: Z1 +- W X
D. Dimension: datt x d
7: W f- LayerNorm(tanh(WW) Dimension: na
7: Z2 + Wate
Dimension: dat x
8: Q t Wh.
8: W Z Zi
Dimension: d x
9: W + LayerNorm(tanh
10: y Wh,
D Dimension: d
11: Q + RMSQ(U)
Figure 3: Comparison of AD and ARQ implemented on top of AD. For ARQ, action conditioning applied part of the input (Line 5,6, Algorithm 2). Note that ARQ allows Zz and y to have dimension (red), which can be arbitrary, while AD fixes it at n, (blue), one for each action output.
Why ARQ benefits local Q-lcarning? As demonstrated in Figure 3 ARQ allows the hidden output have arbitrary dimensions. We conjecture that ARQ's flexibility account for arbitrary hidden dimensions allows it to take full ad advantage of non-linearity within each AD cell. Furthermore, ARQ applies action conditioning at the model input, rather than using vector indices the output layer as conditioning. We conjecture that this allows the entire module to produce representation specific each state-action pair, rather than aggnosic information based on only the observation. Combining these two properties, ARQ exploits the full capacity of the attention-like mechanism that modern local RL methods operates on, allowing greater expressivity of each state-action pair.
EXPERIMENTS
Benchmarks: We test ARQ on the MinAtar benchmark Young & Tian 2019) and the DeepMind Control (DMC) Suite (Tassa al.) 2018) following Guan al. (024). MinAtar miniaturized version of the Atari 2600 games, using T0x10 grids instead of 21 Ox 160 frames inputs. The DMC Suite is a benchmark for continuous control tasks featuring lowelevel observations and actions, de- signed to evaluate the performance of RL methods in physics-based environments. Both benchmarks involve low-dimensional inputs and outputs instead of high-dimensional raw sensory inputs, mak- ing them appropriate testbeds for evaluating the decision-makin ability of local methods in simple environments.
Baselines: For comparisons with cutting- eegie local RL methods, we compare results with AD for both benchmarks. To evaluate our methods against backprop- algorithms, we also com- pare method against DQN for MinAtar. DQN is widely used baseline that trains deep neural

---

Published as a conference paper at ICLR 2026
Freeway
Breakout
SpaceInvaders
Seaquest
Asterix
60
90
600
30
60
400
20
DON
30
200
40
A0
0
ARQ 10uns)
0
2
2
0
3
2
1e6
1e6
1e6
1e6
1e6
Figure 4: Training performance on the MinAtar games, compared between DQN (blue), AD (red), and ARQ (green). The x-axis denotes the number of training steps (in millions), and the y-axis indicates average episodic returns. Shaded regions represent 95% confidence intervals across seeds. We find that ARQ consistently outperforms AD in all MinAtar games, while outperforming DQN in some games.
networks to directly compute scalar Q-values through backpropagation. We follow the DQN imple- mentation used by Guan et al. 20244.
Implementation Details: Following Guan et al. 2024), we use a three-layer fully-connected net- work, with hidden dimensions being 400, 200, and 200 for MinAtar. We use three-layer network with hidden dimensions 128, 96, and 96 for DMC tasks. We use replay buffer and target network for stability. We incorporate skip connections from the input and top-down connections from the layer above. For all experiments, we use an epsilon-greeds policy with linear decay from 1 to 0.01 using an exploration fraction of 0.1. We run our experiments for million steps, where the model starts learning from step 50,000. Learning rate is set fixed at le-4. A batch size of 512 is used. For MinAtar, we condition on action candidates by passing them as one-hot vectors into the network. For DMC tasks, we discretize our action space and condition action vectors model inputs.
As presented in Figure 4 we run each experiment with five different random seeds and plot their average returns over 100-episode windows along with their 95% confidence intervals in shadows. We also calculated the average returns of the last 100 episodes of each training run to obtain a quantitative measure of the final performance of our method, which can be found in Table I As demonstrated, ARQ consistently outperforms AD in all MinAtar games. Surprisingly, ARQ also outperforms DQN in all games. In DMC Suite tasks, ARQ achieves superior returns compared to AD, while also exceeding back-prop based methods in most games. A possible explanation is that ARQ benefits from localized TD updates, reduced gradient path length, and the variance reduction effect of layerwise averaging, which together can lead to more stable and efficient learning than fully backpropagated networks.
Main Results:
We note that ARQ outperforms DQN by wide margin on Breakout and SpaceIn- vaders. Both of these games operate on similar mechanisms: players aim to remove targets by controlling projectile interactions of objects. To yield higher scores, players need to perform com- bos of actions to yield higher scores, for instance moving to a sweet spot then waiting for the target to arrive before firing bullet. We argue that top-down connections in AD provide temporal coherence, which allows our agents to perform sequences of actions smoothly. Additionally, we note that while AD fails to match DQN Seaquest, ARQ surpasses DQN. Seaquest is game involving firing bullets to remove enemies, with an additional rule that players need to manage an oxygen tank by surfacing above water refill their tank. This represents that the policy distribution can be bi-modal such that attacking enemies and refilling tanks are both locally optimal policies. We hypothesize that by applying action conditioning, ARQ can capture these policy structures more effectively than AD, which is only saate-conditione..
Game Analysis:
How does action conditioning affect the performance of local RL methods? To investigate, we conduct ablation experiments on two games from MinAtar, Breakout and SpaceInvaders, using both AD and ARQ. The results can be found in Figure We find a significant improvement when actions are conditioned at the input instead of at the output. To fur- ther understand this difference, we analyzed the hidden activations using PCA and compared them against predicted Q-values as presented in Figure 6 Without action conditioning, activations cluster almost entirely by action identity and show no meaningful correlation with Q-values, indicating that action-related variance dominates the representation space. With action conditioning, representa- tions become more state-driven and exhibit a mild positive relationship with Q-values, suggesting that the model can allocate capacity toward value-relevant structure rather than implicitly inferring
Effect of Action Conditioning Input:
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance of previous methods and ARQ on MinAtar and DeepMind Control (DMC)
tasks. Reported as mean + 95% confidence intervals across 5 random seeds.
MinAtar
Freeway
Breakout
Spacelnvaders
Seaquest
Asterix
w/ back-prop
DQN
55.86
0.64
27.09 +: 11.48
188.03 + 31.62
37.96
18.56
13.60 1 2.16
w/o back-prop
AD
57.12 2.70
63.76 1 17.70
363.49 1 36.92 544.99 1 88.10
27.83
10.97
22.01 1 10.26
ARQ (Ours)
60.74 1 0.54
87.84 1 11.10
96.45
21.44
35.32 1 5.33
DMC
Walker Walk
Walker Run
Hopper Hop
Cheetah Run
Reacher Hard
w/ back-prop proppprop
TD-MPC2
958.80 + 2.58 980.43 + 3.26
834.07 + 20.26
348.55 + 53.30 319.46 + 62.42
808. 46 + 184.20
934.84 + 13.72 980.01 + 2.38
SAC
895.02 + 92.70
917.40 +
4.90
w/o back-prop
AD
975.30
2.10
762.51 + 4.86
470.95 + 78.14
831.57 + 31.80 880.61
955.93 + 18.36
ARQ (Ours)
976.33 1.04
771.15 + 5.08
516.23
34.72
24.04
973.66 + 9.98
Breakout
Spacelnvaders
90
AD w/ AC
600
AD
60
400
ARQ WiO AC
30
200
0
1
2
3
0
1
2
3
4
1e6
1e6
Figure 5: Ablation on action conditioning for AD and ARQ. Action conditioning substantially im- proves performance. Shaded regions represent 95% confidence intervals. Note that this improve- ment is particularly significant for ARQ, with average returns of ~85 vs. ~55, 50% improvement. This indicates that the combination the RMS function and action conditioning makes ARQ effec- tive.
action identity. Interestingly, this design choice provides only slight improvement for AD, while yielding significant increase in performance for ARQ. We conjecture this is due to the increase in the capacity of each cell to capture the granularity within each specific state-action pair, while AD saturates with action-agnostic information.
Effect of Goodness Nonlinearities: One ques- tion that naturally arises is the choice of the goodness function. Does the RMS function perform superiorly compared to other func- tions? We ablate on this design choice and conduct experiments nAtar, Breakout and SpaceInvaders. As shown in Table 2, we find that using the RMS good- ness functions yields superior performance, fol- lowed by the mean and the mean squared func- tion. Beyond performance, our analysis sug- gests that RMS maintains healthier activation magnitudes throughout training, whereas us- ing mean squared function produces extremely large early goodness values that later suppress AD
Table 2:
Our method Using Different Nonlinear- ities Compared in MinAtar Breakout. 'MS' is short for the mean squared function and 'Var' is short for variance. Default ARQ uses the root mean squared (RMS) function. Reported as mean 95% confidence intervals.
games from Mi-
Nonlinearity
Breakout
SpaceInvaders
Ours-ARQ
87.84+111.10
544.99 18.10
Ours-Mean
79.84422.46
500.13:495.56 56
Ours-MS
82.10+16.56
434.88+-28.74
Ours- Var
81.341 0.78
416.46# 133.2
67.40 8.02
369.96 + 46.92
activation norms (see Figure 7. This stabiliza-
tion effect likely preserves richer and more expressive representation space, contributing RMS's empirical advantage. However, we note that all functions perform superiorly compared with AD, which demonstrates the versatility of our method.
Is it because ARQ has more hidden units? Compared with AD, ARQ employs larger number of parameters since ARQ allows arbitrary dimension for its hidden vectors. Could ARQ, however,
8

---

Published as a conference paper at ICLR 2026
ARQ
ARQ w/o AC
PCA Component 2
PCA Component
-10
450
-26
460
200
PCA Component
PCA Component
NOOP
08
LEFT
RIGHT
Q-valuie
Q-Value
-100
PCA Component
-20
PCA Component
30
200
Figure 6: Representation analysis of ARQ with and without action conditioning. We train two agents ARQ with and without action conditioning (AC) on MinAtar Breakout, then randomly sample 200 states from each trained policy. For each state action pair, we extract the hidden acti- vations from Layer 0 and visualize them using 2-component PCA. The top row shows how mPCA components cluster across different actions. Without AC, activations form tight clusters determined almost entirely by action identity, indicating that the agent must implicitly encode action informa- tion inside the representation. With AC, the activations bottom row plots the first PCA component against the predicted Q-values. With AC, ARQ exhibits mild positive correlation between latent structure and Q- Q-values, while the non-AC model shows no meaningful correlation and remains dominated by action-specific clustering.
more entangled and state-driven. The
simply achieve the same improvement with mere scaling? We conduct experiments on AD and ARQ with the same number total parameters to answer this question. Across different ratios of total parameters (compared with the original AD baseline), MinAtar Breakout game with two different random seeds. As shown in Table 3 ARQ consistently outperforms AD across all scales. This verifies the effectiveness of our method beyond scale.
both AD and ARQ the
Neurons Are Sensitive to Different Scenar- ios: How does method leam through goodness function? We investigate its inner mechanism by visualizing the activations at each layer under different states. As illustrated Figure g we find that the hidden activations tend to show larger magnitudes under "correct" state-action pairs. For instance, scenarios where the agent should move right to accurately catch the incoming ball, neurons in the hidden activations show the largest magnitude when the action input matches correspondingly. In- terestingly, observe that different neurons in general, activated to different degrees for vari- ous action candidates. This implies our objective function could be encouraging specialized neurons, each of which is responsible for recognizing certain categories of positive signals.
Table 3: AD vs. ARQ Across Multiple Scales for MinAtar Breakout. Reported as mean 95% con- fidence intervals.
Scale Ratio
AD
ARQ
0.5x
66.34 + 5.15
68.12 + 5.65
1x
64.20 + 1.90
86.26 + 0.66
1.5x
56.63 + 5.39
70.40 = 3.98
2x
59.79 +: 4.77
83.26 2.32

---

Published as a conference paper at ICLR 2026
Step 0
Step 3999999
Activation Magnitude (L2 Norm)
Activation Magnitude (L2 Norm)
ARQ
ARQ
ARQ-MS
ARQ-MS
Density
Density
1455
L2 Norm Hidden Activations
Norm Hidden Activations
Goodness Value Distribution
Goodness Value Distribution
ARQ
ARQ ARQ-MS
ARQ-MS
Density
Density
Goodness Value
Goodness Value
Figure 7: Analysis of ARQ (RMS goodness) -Atar-Breakout. We evaluate hidden activations and goodness values of both agents over 200 ran- domly sampled states both the start and end of training. Top row: Distribution of 1.2 norms of hidden activations. Early in training, ARQ-MS exhibits extremely large goodness values and com- pressed activation magnitudes, while ARQ maintains broader, more expressive activation scales. By the end of training, ARQ continues to support significantly larger activation norms, whereas ARQ- MS activations remain narrowly concentrated. Bottom row: Distribution of scalar goodness values derived from each hidden vector. ARQ produces moderate, stable goodness magnitudes throughout training, while ARQ-MS shows large initial spikes followed by sharply reduced variability.
ARQ-MS (mean-squared goodness) Mi-
DISCUSSION
Previous studies on biologically plausible learning have largely focused on the search for biologi- cally plausible mechanism for performing gradient updates. As we approach the era of experience, argue that biologically plausible paradigm for learning can be equally meaningful to guide us towards the mystery behind how biological brains learn. Reward-centric environments provide a bi- ologically grounded paradigm, aligning with the evolutionary role of survival signals and behavioral shaping through positive or negative reinforcement. The structure of such environments mirrors the ecological settings in which animals adaptively refine behavior through trial-and-error interactions, suggesting that learning systems shaped by rewards may naturally emerge in both artificial and bio- logical agents. Additionally, temporal difference methods that biological neurons learn through temporal difference, with hormones conveying the prediction error as source of learning signal to independent neurons (Schultz et al., 1997b) Bayer Glimcher 2005). On the other hand. reinforcement learning has largely focused on learning through interac- tions with an agent's surrounding environment, and maximizing its rewards through centralized value estimation. Yet, increasing neuroscientific evidence has shown that neurons make decentral- ized, independent value estimations (Tsutsui et al. 2016 Knutson et al. 22055.. Few work the RL community has investigated whether this biological phenomenon has practical implications. ARQ an effort towards this direction as each cell in our network can be seen as a decentralized value estimator.
ideal candidate. It has been shown
7 CONCLUSION
This work proposes Action-conditioned Root mean squared Q-Function (ARQ), a vector-based al- ternative to scalar Q-learning for backprop-fiee free local learning. ARQ enables arbitrary hidden di- mensions and improved expressivity by extracting value predictions from hidden activations and applying action conditioning the model input. We show that, when applied on RL environments, ARQ performs superiorly compared to current local methods, while also outperforming backprop- based methods some games. Whereas current biologically plausible algorithms mostly based on the supervised setting, our study suggests that exploring local learning within reinforcement learning may provide promising avenue for future research both domains.
10

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
LLMs used in this work refine certain textual phrasing and to generate minor code elements related to visualization and checkpointing.
REPRODUCIBILITY STATEMENT
We provide training code for all experiments supplementary material. All hyperparameters, ran- dom seeds, and implementation details are described in Section [ Our experiments were run on single NVIDIA A4000 GPUs NVIDIA L40S GPUs with training times ranging from 8-72 hours depending on the task. We used only publicly available benchmarks (MinAtar and DeepMind Con- trol Suite). Together, these resources should enable full reproduction of our results.
ACKNOWLEDGEMENT
We thank Jonas Guan for his help in reproducing AD. MR supported by Visko Al and the In- stitute of Information & Communications Technology Planning Evaluation (ITP)) under grant RS- 2024-00469482, funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research. The compute is supported by the NYU High Performance Computing resources, services, and staff expertise.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: 1806. 06920,, 2018.
Ryunosuke Amo, Sara Matias, Akihiro Yamanaka, Kenji Tanaka, Naoshige Uchida, and Mitsuko Watabe-Uchida. A gradual temporal shift of dopamine responses mirrors the progression of tem- poral difference error in machine learning. Nature neuroscience, 25(8): 1092, 2022.
Gürdal Arslan and Serdar Yüksel. Decentralized q-learning for stochastic teams and games. IEEE Transactions Automatic Control, 62(4):1545-1558 2016.
Atilim Günes Baydin, Barak A. Pearlmutter, Don Syme. Frank Wood, and Philip H. S. Torr. Gradi- ents without backpropagation. CoRR, abs/2202.08587, 2022.
Hannah M Bayer and Paul W Glimcher. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1):129-141, 2005.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In Proceedings of the 36th International Conference on Machine Learning, ICML, 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference machine learning, pp. 1096- 1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings the AAAI conference artificial intelligence, volume 32, 2018b.
Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:20/ 1.09533, 2020.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference machine learning, pp. 1146-1 155. PMLR, 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv: /706. 10295, 2017.
11

---

Published as a conference paper at ICLR 2026
Louis Fournier, Stéphane Rivaud, Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Can forward gradient match backpropagation? In International Conference on Machine Learning, pp. 10249-10264, PMLR, 2023.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Mare Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learn- ing. arXiv preprint arXiv: 1704. 04651, 2017.
Jonas Guan, Shon Verch, Claas Voelcker, Ethan Jackson, Nicolas Papernot, and William Cunning- ham. cemporal-differnece learning using distributed error signals. Advances in Neural Informa- tion Processing Systems, 37: 37:108710-10873: 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming. pp. 1861-1870. Pmlr, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.0555 18/2.05905, 2018b.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01600 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International conference on machine learning, pp. 2555-2565. PMLR, 2019b.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023. URL https://arxiv.. org/ubs/2301.04445 041044 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control. arXiv preprint arXiv:2310. 16828, 2023.
Matthew J Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI fall symposia, volume 45, pp. 141, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan
Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arX1v:2212.13345, 2(3):5, 2022.
Francesco Innocenti, Mehdi Achour, and Christopher Buckley, μpe: Scaling predictive coding to 100+ layer networks. arXiv preprint PrXiv:2505.13124 2025.
Jiechuan Jiang and Zongqing Lu. Best possible q-learning. arXiv preprint arXiv:2302.0/ 188, 2023.
Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning-a a simple, efficient, decentral- ized algorithm for multiagent rl. arXiv preprint 15X15:2110.14555, 2021.
A. Harry Klopf. The Hedonistic Neuron: A Theory of Memory, Learning and Intelligence. Wash- ington: Hemisphere Pub. Corp., 1982.
12

---

Published as a conference paper at ICLR 2026
Brian Knutson, Jonathan Taylor, Matthew Kaufman, Richard Peterson, and Gary Glover. Distributed neural representation of expected value. Journal of Neuroscience, 25(19):4800 0104480064812,, 2005.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Martin Lauer and Martin A Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In Proceedings of the seventeenth international conference on machine learning, pp. 535-542, 2000.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wrxiv:1509002977 2015.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1); 13276, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PmLR, 2016.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In Inter- PMLR, 2019.
national conference on machine learning, pp.
John P O' O'Doherty, Peter Dayan, Karl Friston, Hugo Critchley, and Raymond J Dolan. Temporal difference models and reward-related learning in the human brain. Neuron, 38(2):329-337, 20003299333,,003 2003.
Alexander Ororbia and Ankur Mali. The predictive forward-forward algorithm. arXiv preprint arXiv:2301.01452, 2023.
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep rein- forcement learning. arXiv preprint arXiv:1707.04402, 2017.
Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. Convolutional channel-wise competitive learning for the forward-forward algorithm. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp.
2024.
Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In ICLR, 2023.
Martin Riedmiller. Neural fitted q iteration -irst experiences with data efficient neural reinforce- ment learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6089)5333-536 1986.
Jacob Sacks, Rwik Rana, Kevin Huang, Alex Spitzer, Guanya Shi, and Byron Boots. Deep model predictive optimization. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 16945-16953. IEEE, 2024.
Julian Schrittwisser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with leamed model. Nature, 588(7839)6604-666 2020.
13

---

Published as a conference paper at ICLR 2026
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 0243888,, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, (7555306)11533-1553 1997a.
Wolfram Schultz, Peter Dayan, and Read Montague. A neural substrate of prediction and reward. Science, 275(5306):1593-555533 1997b.
Tim Seyde, Igor Gilitschenski, Wilko Schwarting. Bartolomco Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems,
2021.
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via q-learning. arXiv preprint arXiv:2210. 2022.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwesser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844889 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 1rXiv:17/2.018855, 2017.
Utkarsh Singhal, Brian Cheung, Kartik Chandra, Jonathan Ragan-Kelley, Joshua B Tenenbaum, Tomaso A Poggio, and Stella Yu. How guess gradient. arXiv preprint arXiv:23/2.04709 2023.
Kefan Su and Zongqing Lu. f-divergence policy optimization in fully decentralized cooperative marl.
Kefan Su and Zongging Lu. Decentralized policy optimization. arXiv preprint arXiv:2211.03032. 2022.
Kefan Su, Siyuan Zhou, Jiechuan Jiang, Chuang Gan, Xiangjun Wang, and Zongqing Lu. Ma2ql: A minimalist approach fully decentralized multi-agent reinforcement learning, arXiv preprint arXiv:2209.088 08244, 2022.
Liang Sun, Yang Zhang, Weizhao He, Jiajun Wen, Linlin Shen, and Weicheng Xie. Deeperforward: Enhanced forward-forward training for deeper and better performance. In The Thirteenth Interna- tional Conference Learning Representations, 2025. URL htttpp:://ppenreiiee...
Eorum? -
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44, 1988.
Ming Tan. Multi-agent reinforcement learning: independent versus cooperative agents. In ICML/933 Proceedings of the Tenth International Conference on International Conference on Machine Learning. pp. 330-337, 1993.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint arXiv: /801. 000990 2018
14

---

Published as a conference paper at ICLR 2026
Niccolò Tosato, Lorenzo Basile, Emanuele Ballarin, Giuseppe de Alteriis, Alberto Cazzaniga, and Alessio Ansuini. Emergent representations networks trained with the sawaadd-frwald algo- rithm, 2023.
Ken-Ichiro Tsutsui, Fabian Grabenhorst, Shunsuke Kobayashi, and Wolfram Schultz. A dynamic code for economic object valuation in prefrontal cortex neurons. Nature communications, 7(1):
12554, 2016.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Ziyu Wang, Tom Schaul, Mateo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Mitsuko Watabe-Uchida, Neir Eshel, and Naoshige Uchida. Neural circuitry of reward prediction eπror. Annu. Rev. Neurosci., 40:373-394, 2017.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2 2021.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 144:25476-25885 2021.
Kenny Young and Tian Tian. MinAtar: An Atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176 2019.
15

---

Published as a conference paper at ICLR 2026
A VISUALIZATION OF ARQ ACTIVATIONS
Figure X
provides the full activation visualizations referenced in the main text, illustrating how
neuron responses vary across different state-action scenarios in Breakout.
State (Left Action)
Selected Neuron Activations
Activation Value
Global Avg Neurons)
Left Action
Move Left
Right Action
Root Mean Squared Activations
Move Right
Selected Neuron Index
State (Right Action)
Selected Neuron Activations
Activation Value
Left Action
Right Action
Avg Activation (All Neurons)
Selected Neuron index
Selected Neuron Index
Figure 8: Visualization of Neurons in Layer O under Different Scenarios in Breakout Game. 20 neurons w/ highest average activities are visualized. Top Left: When the ball is approaching towards the left side of the brick, neurons show larger magnitude when the action candidate is to "move left", prompting the agent to move towards the ball. Bottom Left: When the ball approaches the right side of the brick, neurons show larger magnitude when the action candidate is to "move right"" Right: The average root mean squared (RMS) activations of 20 top neurons across 100 states collected. Note that top neurons exhibit significantly larger RMS activations than the average RMS activation, implying that these ncurons are "dominant" neurons. While most neurons demonstrate similar magnitude between both actions, some neurons appear to be more specialized.
B HYPERPARAMETERS
We summarize all hyperparameters used for MinAtar and DeepMind Control Suite experiments in Table 4 consolidating the network architectures, training settings, and optimization details for full reproducibility.
Hyperparameter
MinAtar
DMC Suite
Network Architecture
3-layer MLP 400-200-200
3-yayer MLP 128 96-96 Adam 0.99
Hidden Dimensions Optimizer
Adam
Discount Factor (7)
0.99
Learning Rate
x 10-4
x 10-4
Batch Size
512 4M transitions
512
Replay Buffer Size
4M transitions
Target Network
Exploration Strategy
e-greedy
e-greedy (on discretized actions)
Schedule
Linear: 1.0 - 0.05 0.1
Linear: 1.0 - 0.05
Exploration Fraction
0.1
Learning Starts
50,000 steps
50,000 steps
Training Steps
4M
4M
Action Representation
One-hot
Bang- bang discretization
Random Seeds
Table Consolidated hyperparameters for MinAtar and DeepMind Control (DMC) experiments. All settings follow the implementation described in Section 5 of the paper, with Adam optimization, a shared replay buffer of 4M transitions, and identical training schedules. Action conditioning is applied for ARQ by default.
16

---

Published as a conference paper at ICLR 2026
C DETAILED COMPUTATIONAL DIAGRAM
In this section, provide detailed visualization of the computation flow in both AD and ARQ, illustrating how information propagates across layers and how each cell locally estimates Q-values.
Artificial Dopamine (AD) (Guan et al., 2024)
[na)
[d. na
hfd
tanh
News
Wau
W/
hea2
x; 11.1
Ri-1
Ag
x; Ry
Riws
R2=2
Kews
ARQ (Ours)
(x)
R
B(-}
[d]
w [d, d]
hg
tanh
relu
Watt
W₂
hersi
Sel R-1
Figure 9: Detailed computation diagram. Key implementations of ARQ are highlighted in red. Left: Overall information processing across layers. Each cell receives input from the raw observation, layer below, and the layer above, and its activations Top Right: Cell Computation in Artificial Dopamine (AD); na Q-value estimates are produced by dot-products between the attentional weights and the hidden states. Bottom Right: Our ARQ implements root mean squared functions for value estimation along with action-conditioneed inputs.
passed forward to the next temporal step.
D DEEPMIND CONTROL SUITE EXPERIMENTAL FIGURES
presents the full training curves for all DeepMind Control Suite environments. ARQ consistently outperforms AD across tasks and achieves performance comparable to strong backpropagatton--base methods such as TD-MPC2 and SAC.
Figure
TO
Walker Walk
Walker Run
Hopper Hop
Cheetah Run
Reacher Hard
900
900
900
900
400
600
600
600
AD
300
TOMEC2
ARD (Oun)
300
300
0
0
0
2
3
0
2
3
0
2
3
0
3
1
2
4
1e6
le6
1e6
1e6
1e6
Figure 10: Training performance on the DeepMind Control Suite, compared between AD (red), ARQ (green), TD-MPC2 (blue), and SAC (gray). The x-axis shows training steps (in millions), and the y-axis denotes average episodic returns. Shaded regions indicate 95% confidence intervals across seeds. Across all environments, ARQ consistently improves over AD and achieves performance competitive with backpropagation based methods.
17

---

Published as a conference paper at ICLR 2026
E NONLINEARITY ABLATION FIGURES
Figure 11 presents the per-environment learning curves comparing different nonlinear goodness functions used to extract scalar values from hidden vectors. The default ARQ, which uses the root mean squared function, outperforms all other choices. Notably, all variants outperform AD in both settings.
Breakout
Spacelnvaders
90
ARQ-MS
ARQ-Mean
500
60
ARQ
ARQ Var
250
30
AD
1
4 1e6
1e6
Figure 11: Comparison of different nonlinear goodness functions that may be used to collect scalar statistics from hidden vectors. 'MS' short for the mean squared function and "Var' is short for variance. Default ARQ uses the root mean squared (RMS) function. Shaded regions represent 95% confidence intervals.
18


Reviews:
[{"review_id": "16jFlGxO62", "reviewer": "Reviewer_KK5z", "summary": "This paper introduce an ARQ method which does not use the backpropagation update. Experiments on MinAtar benchmarks show ARQ has higher reward than backpropation methods.", "questions": "1. The authors note that output-layer conditioning is computationally efficient for discrete tasks with low-dimensional action spaces. Could you provide quantitative or qualitative evidence on how ARQ compares in terms of computational efficiency to previous local RL methods (e.g., AD) and backprop-based baselines? For example, does ARQ reduce memory usage, training time, or inference cost?\n\n2. Is it feasible to integrate ARQ into an actor–critic architecture? If so, what are the main challenges or design constraints that would need to be addressed?\n\n3. Could you clarify the conceptual difference between decentralized Q-learning (as in ARQ or multi-agent setups) and distributional Q-learning approaches like D4PG? While distributional Q models the return distribution, does it also imply decentralization, or are these fundamentally distinct paradigms?\n\n4. What are the shades in Figure 5 represents for? is it std or confidence interval? Please be specified in the caption", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the idea is novel as it bridges FF and TD learning\n2. the result looks promising. Experiments on MinAtar benchmarks show ARQ has higher reward than backpropation methods.", "weaknesses": "1. The author should carefully review the manuscript for consistency and completeness. For example: Line 156: The term “q-function” should be standardized to “Q-function” for consistency throughout the paper. Line 342: The placeholder “Figure ??” needs to be replaced with the correct figure number.\n\n2. Based on Table 2, while ARQ demonstrates strong performance compared to local RL methods and even surpasses backprop-based algorithms in many discrete tasks, its results on continuous control tasks (DeepMind Control Suite) remain slightly below state-of-the-art actor–critic methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8eYqwTgbmZ", "reviewer": "Reviewer_xkdR", "summary": "This work follows a recent trend of modifying neural network training to operate in a layer-by-layer manner, rather than processing inputs through all layers and updating the network parameters in a single pass. It builds on Artificial Dopamine (AD), which learns Q-values by training each layer separately and then averaging the value estimates from all layers during policy execution. This work modifies the AD architecture by incorporating the action directly into each layer’s input, whereas AD only takes the state as input and outputs multiple action-specific heads—an approach more similar to a DQN-style network.\n\nThe authors conducted comprehensive experiments across a diverse set of tasks, demonstrating a consistent performance improvement of their method, particularly over its counterpart, Artificial Dopamine.", "questions": "A specific question regarding the architectural design:\n\nAs far as I understand AD, each layer contains multiple attention matrices, with the number matching the size of the discrete action space. Specifically, for each layer $l$, there is a set of matrices $W_{\\text{attn},1}^{[l]}, W_{\\text{attn},2}^{[l]}, \\ldots, W_{\\text{attn},|A|}^{[l]}$, where each is responsible for computing the Q-value corresponding to a particular action.\n\nIn this work, the authors argue that they remove this constraint. In particular, according to Equation (13), the network outputs a vector $y_t^l$. However, it is unclear to me how such a vector could be generated using a single attention matrix, assuming $W_{\\text{attn}}$ is a two-dimensional matrix and $\\mathbf{X}$ and $\\mathbf{h}_t^l$ are vectors.\n\nIs it the case that the authors introduce an arbitrary number of attention matrices to produce the vector $\\mathbf{y}_t^l$? Could you provide more details on the architectural design so that the innovation at the architecture level becomes clearer?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed modification represents a solid and reasonable extension to AD, supported by empirical results that show clear improvements over the original method. Additionally, the authors provide comprehensive ablation studies that further validate the effectiveness of their approach.", "weaknesses": "I would suggest that the authors consider running experiments with additional random seeds to more robustly validate the advantages of the proposed method over AD. Using only three seeds—especially when reporting such large performance gaps—may not be sufficiently convincing to support the claim that the proposed approach is fundamentally superior.\n\nMoreover, this line of research is largely driven by heuristic intuitions inspired by analogies between biological and artificial neurons. Given the substantial performance improvements observed, the contribution would be significantly strengthened by providing deeper theoretical insights, particularly in the context of reinforcement learning. For instance, value estimation in the Bellman equation can be viewed as a dynamic programming process that propagates rewards across state–action pairs. Does the proposed method accelerate this propagation, or is there another underlying mechanism that explains the improvements? A clearer theoretical justification—beyond empirical evidence—would help reinforce the contribution and clarify why both the proposed method and its predecessor, AD, achieve such notable gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wuOWKmgr3e", "reviewer": "Reviewer_QJn3", "summary": "The paper introduces ARQ, a biologically inspired local learning algorithm that extends Hinton’s Forward-Forward paradigm to reinforcement learning. ARQ replaces backpropagation with a forward-only value estimation mechanism using action-conditioned root mean squared Q-functions, enabling local, backprop-free RL. The work looks to extend the Artificial Dopamine framework by eliminating some action-space constraints. The results show some improvements on MinAtar and DM Control Suite benchmarks.", "questions": "While the analogy to the “goodness” function from the Forward-Forward algorithm is intuitive, there’s no mathematical argument explaining why this RMS formulation should lead to better learning stability or generalization. Can you offer insight?\n\nThe claimed innovation — using the root mean squared of the hidden vector after mean subtraction — is a relatively minor modification to existing FF or AD approaches. It’s more of a functional reparameterization than a conceptual breakthrough, and the improvement in results could partly stem from increased capacity rather than a new idea.\n\nIt remains unclear whether ARQ could handle high-dimensional, visual, or partially observable tasks.\n\nAblations are limited to choice of nonlinearity and action conditioning. There is little exploration of architectural sensitivity (e.g., hidden dimension size, number of layers, effect of top-down connections and so on). Given the absence of theory, a more extensive empirical analysis would strengthen the paper.\n\nJust 3 seeds must give poor measures of standard error – this just seems wrong… you then go on to bold best performing algo in the tables, without considering statistical significance – even when there are only 3 replicates and stds overlap. Canyou give significance?\n\nMinor things\nP7 Figure ?? – latex compile issue\nProtect Capitals in bib – like {A}tari", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and written in a clear, accessible manner. The motivation, extending Hinton’s biologically plausible FF learning to reinforcement learning, is well introduced. Figures effectively illustrate the difference between supervised and local RL setups and the architecture of ARQ versus Artificial Dopamine baseline. \n\nThe experiments are conducted on standard small-scale with consistent settings so comparisons against AD baseline and backprop-based methods like DQN and SAC are appropriate. \n\nAcross both benchmarks, ARQ consistently outperforms AD and can be better than traditional backprop-based methods. \n\nThe visualization of neuron activations provides a nice illustration of how the proposed method might be creating local representations for specific state-action pairs.\n\nThe biological plausibility of the approach is discussed.", "weaknesses": "Conceptually, the paper is a pretty straightforward extension of Guan et als 2024 Artificial Dopamine work. The core novelty seems to be in replacing the scalar Q-value with an RMS alternative. \n\nThere is little extension of the AD idea for biologically plausible RL. \n\nMuch of the architectural structure, training pipeline and evaluation setup is directly taken from the AD work\n\nThe root-mean-square formulation is introduced heuristically with little theoretical motivation or analysis of properties – which means the paper lacks new theoretical insight or learning principle beyond empirical observation.\n\nThe evaluation is restricted to low-dimensional benchmarks. These are useful for proof-of-concept, but they are not sufficient to demonstrate scalability or robustness of the approach. \n\nThe paper repeatedly emphasizes biological plausibility, but the actual learning rule remains a standard TD-style update with neural architectures that use FF’s local updates. There’s no evidence of truly biologically grounded mechanisms so biological plausibility feels somewhat overplayed.\n\nWhile the ablation studies are appreciated, they are minimal. limited to choice of nonlinearity and action conditioning. There is little exploration of architectural sensitivity (e.g., hidden dimension size, number of layers, effect of top-down connections). Given the absence of theory, a more extensive empirical analysis would be needed to strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LukGQ81u5j", "reviewer": "Reviewer_WJCj", "summary": "The paper proposes an extension of the recently developed artificial dopamine framework for local reinforcement learning (RL with no or limited backpropagation). The authors propose to use the action as input to the local networks Q prediction, instead of parameterizing a vector of per-action Q function, as done in AD and related RL works such as DQN> They show the impact of their change in empirical experiments on common small-scale RL benchmarks.", "questions": "Could other forward-forward architectures such as Sun et al. 2025 also be used to improve AD?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The change to the previous AD setup is small, but impactful (I consider small clear changes a strength!). The experiemnts clearly show that moving the action to the input of the network can yield large performance gains. The writing is clear, and the experiments are reasonably designed.\n\nOverall, I believe the method here has merit, but it's current presentation is a bit too superficial for me to fully endorse acceptance (see below). At the same time I believe there are no reasons the paper should not be accepted.", "weaknesses": "While the authors investigate the impact of the action-conditioning, the additional experiments (e.g. Figure 6) don't serve to enhance the understanding of the concrete change proposed in this paper. Why does moving the action to the input conditioning help compared to a DQN like parameterization? This is currently my main issue prevent me from giving a higher rating to the paper and I would be happy if the authors addressed this in the rebuttal.\n\nThe papers usefulness for future authors would be helped with a standard appendix containing per env training curves for all algorithms, as well as a clean diagram of the architecture and an easy reference table for hyperparameters. This is standard in the RL literature, and as such it would help the authors method reach a wider audience.\n\nThe performance differences between AD and ARQ in the DMC seem partially insignificant, yet they are highlighted. Please ensure that confidence metrics are accounted for.\n\nThere are some missing references and other polishing errors:\n- Line 342: Figure ??\n- Figure 5: ARQ and ARQ w/o AC are indistinguishable", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)