Spatial-Aware Decision-Making with Ring Attractors in Reinforcement Learning Systems
Authors: Marcos Negre Saura
Richard Allmendinger
Wei Pan
Theodore Papamarkou
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Computational Neuroscience
Deep Learning
Ring Attractors
Spatial Awareness
Bioinspired
Primary area: reinforcement_learning
TL;DR: Integrating neuroscience-inspired ring attractors into Reinforcement Learning action selection process.

Abstract: Ring attractors, mathematical models inspired by neural circuit dynamics, provide a biologically plausible mechanism to improve learning speed and accuracy in Reinforcement Learning (RL). Serving as specialized brain-inspired structures that encode spatial information and uncertainty, ring attractors explicitly encode the action space, facilitate the organization of neural activity, and enable the distribution of spatial representations across the neural network in the context of Deep Reinforcement Learning (DRL). These structures also provide temporal filtering that stabilizes action selection during exploration, for example, by preserving the continuity between rotation angles in robotic control or adjacency between tactical moves in game-like environments. The application of ring attractors in the action selection process involves mapping actions to specific locations on the ring and decoding the selected action based on neural activity. We investigate the application of ring attractors by both building an exogenous model and integrating them as part of DRL agents. Our approach significantly improves state-of-the-art performance on the Atari 100k benchmark, achieving a 53\% increase in performance over selected baselines.

Spatial-Aware Decision-Making with Ring Attractors in Reinforcement Learning Systems
Marcos N. Saura
The University Manchester
Manchester, United Kingdom
marcos www.eeaaaaaccppstgrra. manchester...mm ac uk
Richard Allmendinger
The University of Manchester
Manchester, United Kingdom
Richard. An ac.uuk
Wei Pan
Theodore Papamarkou
The University of Manchester Manchester, United Kingdom wei. www.aaaneeecceerr.... ac uk
PolyShape Athens, Greece theodoor@@polyshape..o com
Abstract
Ring attractors, mathematical models inspired by neural circuit dynamics, provide biologically plausible mechanism improve learning speed and accuracy in Reinforcement Learning (RL.). Serving as specialized brain-inspired structures that encode spatial information and uncertainty, ring attractors explicitly encode the action space, facilitate the organization of neural activity, and enable the distribu- tion of spatial representations across the neural network in the context of Deep Reinforeement Learning (DRL). These structures also provide temporal filtering that stabilizes action selection during exploration, for example, by preserving the continuity between rotation angles in robotic control or adjacency between tactical moves in game-like environments. The application of ring attractors in the action selection process involves mapping actions to specific locations on the ring and decoding the selected action based neural activity. We investigate the application ring attractors by both building an exogenous model and integrating them as part of DRL agents. Our approach significantly improves state-of-the-ant perfor- mance on the Atari 100k benchmark, achieving 53% increase in performance over selected baselines.
Introduction
This paper addresses the challenge of efficient action selection in Reinforcement Learning (RL), particularly in environments with spatial structures, ranging from robotic manipulation where joint movements are coupled, to game-playing agents where tactical decisions might depend on positional awareness. We integrate ring attractors, neural circuit model from neuroscience originally proposed by (50) and later experimentally validated by (18), into the RL framework. This approach provides a mechanism for uncertainty-awar decision-making in RL, thereby yielding more efficient and reliable learning in complex environments. Ring attractors offer unique framework to represent spatial information in a continuous and stable manner [33]. In ring attractor network, neurons interconnect circularly, forming a loop with tuned connections [7]. This configuration allows for robust and localized activation patterns, maintaining accurate spatial representations even in the
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

presence of noise or perturbations. Applying ring attractors to the selection of RL actions involves mapping actions to specific ring locations and decoding the selected action based on neural activity. This spatial embedding proves advantageous for continuous action spaces, particularly in tasks such robotic control and navigation (28). Ring attractors improve decision-making by exploiting spatial relations between actions, contributing to informed transitions between actions in sequential decision-making tasks in RI. While many action spaces possess inherent topological structure [39]. traditional neural networks represent actions as orthogonal vectors that ignore these relationships. Ring attractors bridge this gap by providing a neural substrate that preserves spatial ordering and similarity between actions. This is particularly relevant given recent advances showing that spatial structure in representations improves sample efficiency 22 49 and generalization 3 13
yet these
benefits have not been systematically applied to action selection mechanisms themselves.
In what follows, we summarize our contributions. Briefly, our contributions include novel approach to RL policies based on ring attractors, the inclusion of uncertainty- systems, and the development of Deep Learning (DL) modules for RL with ring attractors.
capabilitiess in our RL
We propose a novel approach for integrating ring attractors into RL. incorporating spatial information, temporal filtering, and relations between actions. This spatial awareness significantly speeds up the learning rate of the RL agent. See Sections 331.2 and 4.1
We utilize a Bayesian approach to build ring attractors input signals, encoding uncertainty estimation to drive the action selection process, showing further performance improvement. See Sections
L.
and 4.
We develop a reusable DL module based on recurrent neural networks that incorporates ring attractors into DRL agents. This allows integration and comparison with existing DRL frameworks and baselines, enabling the adoption of our ring attractor approach across different RL models and tasks in applied domains. See Sections 3.2 and 4.2
Codebase available at http:://ggittuub.oommmarrcooooo
2 Related Research
The integration of ring attractors into RL systems brings together neuroscience-insiree models and advanced machine learning techniques. In this section, we review the literature on key areas that form the foundation of our RL research: spatial awareness in RL., biologically inspired RL approaches, and uncertainty quantification methods.
2.1 Spatial Awareness in RL.
Incorporating spatial awareness into RL systems has improved performance on tasks with inherent spatial structure. Regarding relational RI. 49 introduced approach using attention mechanisms to reason about spatial relations between entities in an environment. This method demonstrated improved sample efficiency and generalization in tasks that require spatial reasoning. On the topic of navigation, (25 developed a DRI. agent capable of navigating complex city environments using street-level imagery. Their approach incorporated auxiliary tasks, such as depth prediction and loop closure detection. Concerning explicit spatial representations, [13] proposed cognitive mapping and planning approach for visual navigation, combining spatial memory with a differentiable neural planner. Similarly, B introduced a relational DRL framework using graph neural networks to capture spatial relations between objects. Although these approaches demonstrate the importance of spatial awareness in RL, they rely on emergent learning to develop spatial understanding through compre- hensive architectural additions, which require extensive training to discover action relationships. In contrast, our ring attractor approach explicitly encodes spatial structure in the action space, providing spatial inductive bias that accelerates learning rather than relying on the RL agent to autonomously discover all critical action space relationships, which may result in suboptimal or incomplete spatial understanding.
2.2 Biologically Inspired Machine Intelligence
Biologically inspired approaches to RL seek to leverage insights from neuroscience to improve the efficiency, adaptability, and interpretability of RL algorithms. These methods often draw upon neural circuit dynamics and cognitive processes observed in biological systems. A notable example is

---

the work in 455. which demonstrated how neural attractor states implement probabilistic choices through recurrent network dynamics. The work presented in [2] demonstrated that incorporating grid-like representations, inspired by mammalian grid cells, into RL agents improved performance in navigation tasks. Their work showed that these biologically inspired representations emerged naturally in agents trained on robotics tasks and transferred well to new environments. Similarly, [10] showed that recurrent neural networks trained on navigation tasks naturally developed grid-like representations, suggesting a deep connection between biological and artificial navigation systems. In RL, grid cells and ring attractors serve distinct computational roles. Grid cells encode spatial state information (the agent's position), through hexagonal firing patterns, functioning as fixed basis functions for value functions and spatial generalization 2 10] Ring attractors encode persistent states, directional and spatial bias, through recurrent dynamics that actively maintain information over time [16]. While grid cells provide emergent spatial input features, ring attractors are dynamical components that provide intrinsic spatial encoding for memory states [17]. that here we translate to encoding of the action space. [31 demonstrated how RL agents naturally develop insect-like behaviors and neural dynamics when solving complex spatial navigation tasks, [44] proposed a biologically inspired meta-RL. RL algorithm that mimics the function of the prefrontal cortex and dopamine- based neuromodulation. Their approach demonstrated rapid learning and adaptation to new tasks, similar to the flexibility observed in biological learning systems. However, these approaches primarily focus on mimicking neural representations or learning dynamics, whereas ring attractors specifically encode spatial relationships in the action space itself, providing both biological plausibility and direct spatial awareness for action selection.
2.3 Uncertainty Quantification
Regarding exploration strategies, (27] introduced bootstrapped Deep Q-networks (DONs), addressing exploration by leveraging uncertainty in Q-value estimates by training multiple DQNs with shared parameters. Building on this theme, [ proposed random network distillation (RND), measuring uncertainty by comparing predictions between a target network and a randomly initialized network. For efficient uncertainty quantification, [11] and [9] presented novel "masksemble" approach, applying masks across the input batch during the forward pass to generate diverse predictions. Addressing risk assessment in non-stationary environments, [14] described a method to analyze sources of lack of knowledge by adding a second Bayesian model to predict algorithmic action risks, particularly relevant for multi-agent RL (MARL) systems. [9] developed a Bayesian ring attractor that outperforms conventional ring attractors by dynamically adjusting its activity based on evidence quality and uncertainty. In the context of individual treatment effects, [20) performed uncertainty quantification (UQ) using an exogenously prescribed algorithm, making the method agnostic to the underlying recommender algorithm. I developed a Bayesian approach for RL in episodic high-dimensional Markov decision processes (MDPs). They introduced two novel algorithms: LINUCB and LINPSRL. These algorithms achieve significant improvements in sample efficiency and performance by incorporating uncertainty estimation into the learning process. In the context of DRL. Bayesian Deep Q-networks (BDQNs) [ incorporate efficient Thompson sampling and Bayesian linear regression at the output layer to factor uncertainty estimation in the action- value estimates. While these methods treat uncertainty estimation as a separate module applied to existing architectures, our ring attractor approach integrates uncertainty through the variance parameters of Gaussian input signals, making uncertainty awareness an intrinsic part of the spatial action representation.
In summary, the literature reveals growing interest in incorporating spatial awareness, biological inspiration, and UQ into RL systems. However, there remains a gap in integrating these elements into cohesive framework that simultancously provides biological plausibility, explicit spatial encoding, and natural uncertainty handling within single neural architecture. Our work on ring attractors aims to bridge this gap by providing a biologically plausible model that inherently captures spatial relations and can be extended handle uncertainty, potentially leading to more robust and efficient RL agents.
3 Methodology
Ring attractors represent a computational principle where neurons are arranged in a circular topology to maintain spatial representations. Originally proposed by (50) for encoding heading direction and empirically discovered by [18] in Drosophila, these networks exhibit circular organization where head direction cells encode different spatial directions (38). For our methodology, ring attractors
3

---

integrate multiple cues through Distance-weightee connections [S] 12 and maintain representational stability through excitatory-ine iinibbiorr dynamics 337. This bioinspired structure has been repurposed in this line of research to explicitly encode the action space during the decision-making process. In this section, we describe two main methods: exogenous ring attractor model using continuous- recurrent neural networks (CTRNNs) and a DL-based ring attractor integrated into the RL agent. While the CTRNN model validates the theoretical principles, the DI. architecture eases deployment and adoption in existing DRI. frameworks. Both leverage the ring attractors' spatial encoding capabilities to enhance action selection and performance. In the CTRNN model, we detail the ring attractor architecture, Section 1.1.1 dynamies and implementation, Section 3.1.2] and uncertainty injection, Section 3.1.3] CTRNNs are used for their ability to model continuous neural dynamics and maintain stable attractor states 4 The integrated DL approach offers end-to-end training for efficiency and scalability, detailed in Section 3.2 Ring attractors in RL maintain stable spatial information representations, preserving action relations lost in traditional flattened action spaces. This circular spatial representation potentially yields smoother policy gradients and more efficient learning in spatial tasks, attributed to the ring attractors' ability to maintain a stable representation of spatial information.
3.1
Continuous-Time RNN Ring Attractor Model
During the first stage of the research, the focus is on developing a self-contained exogenous ring attractor CTRNN. This will be integrated into the output of the value-based policy model to perform action selection.
3.1.1 Ring Attractor Architecture
Ring attractors commonly consist of configuration of excitatory and inhibitory neurons arranged in a circular pattern. We can model the dynamics of the ring using the Touretzky ring attractor network 44). In this model, each excitatory neuron establishes connections with all other excitatory neurons, and an inhibitory neuron is placed in the middle of the ring with equal weighted connections to all excitatory neurons.
Excitatory neurons' input signal: Let 2 E R denote the input signal from source number to the excitatory neuron n I,..., N. The total input to neuron n is defined the sum of all input signals for that particular neuron: Un = Ei= she where 2n E R. To model input signals ot of varying strengths, these signals are commonly viewed as Gaussian functions a1 R - R These functions allow us to represent the input to each neuron as a sum of weighted Gaussian distributions. The key parameters of these Gaussian functions are: K,, the magnitude variable for the input signal in index i, which determines the overall strength of the signal; (4; which defines the mean position of the Gaussian curve in the ring for the input signal representing the central focus of the signal; oi the standard deviation of the Gaussian function, which determines the spread or reliability of the signal; and an which represents the preference for the orientation of the neuron in space. These parameters combine the following:
In (aks)
Σ
2i (an)
Σ
I
K, V2TTO
exp
(an - -21)² (-lan
20
1=1
1=1
Neuron activation function: We employ rectified linear unit (ReLU) function f(x) max x hh), where + as activation function for each neuron, where h is a threshold that introduces the non-linear behaviour in the ring.
Excitatory neuron dynamics: The dynamics of excitatory neurons in the ring is described as:
du
A AUR A
Un+A:
Un f(In + en + mm) Un
dt
AL
In Eq. (2) Un represents the activation of the excitatory neuron n, Fn is previously defined in Eq. (1) is the external input to the neuron n of Eq. (T). 6n, E R represents the weighted influence of the other excitatory neurons activation, which is defined mathematically in Eq. 4 The F is the influence from the weighted inhibitory neuron activation to the target excitatory neuron n, and A/ is the time integration constant. This equation captures the evolution of neuronal activation over time, considering both excitatory and inhibitory activations.
Inhibitory neuron dynamics. The activation of the inhibitory neuron, which regulates network dynamics, is described by:

---

du Au di
"+ Al
f(en + mm)
A
Here, u E R represents the inhibitory neuron's activation output, En is the weighted sum of excitatory activations where in this case n is the inhibitory neuron, and f/ R is the weighted self-inhibition activation term. This equation models how the inhibitory neuron integrates inputs from the excitatory population and its own state.
Synaptic weighted connections:
The influence between neurons decreases with distance, modeled by the weighted connections. This weighted connection applies to both excitatory and inhibitory
dl Lmm.)) where d(m,.) |m n| is the distance between neurons and n. For the inhibitory neuron: w = e = = Note that our model contains a single inhibitory neuron placed in the middle of the ring. with a distance of unit to all excitatory neurons. The excitatory (e,,) and inhibitory (7t) weighted connections are also known in the literature as neuron-proximal excitatory and inhibitory voltage or potential. These are then defined as follows:
neurons. For excitatory neurons: (..
(.m-+BB -. Um Wm,
The w wll-Enn
En
Um
m=1
3.1.2 Ring Attractor as Behavior Policy in RL
To integrate the ring attractor model with RL, we need to establish connection between the estimated value of acte-action pairs and the input the ring attractor network. This integration allows the ring attractor to serve as a behavior policy, guiding action selection based on the values learned.
Excitatory neurons' action input signal: We begin by reformulating the input function for target excitatory neuron n from Eq. II. To integrate the ring attractor with RL, we reformulate the input signals by mapping K₃ to the v-value Q(s, a), effectively substituting input signal index i E I with actions A. In other words, the key modification is setting the scale factor K; to the Q-value Q(s, a) of the state-action pair (s, a). that is K, = Q(s,
This formulation is represented in Fig. I and described in Eq. (5) ensures that actions with higher estimated values given more weight in the ring attractor dynamics, naturally biasing the network towards more valuable actions. The orientation of the signal within the ring attractor is determined by the direction of movement in the action space. We represent this as μ4; a. (a), where aa (a) is the angle corresponding to the action a in the circular action space. We define our circular action space A as a subset of R22 where each action E AA is represented by a point on the unit circle. The function a: A + [0, 2 maps each action its corresponding angle on this circle, and a, which presents the preference for the orientation of the neuron n in space. To account for uncertainty in our value estimates, incorporate the variance of the estimated value for each action 0; Fa-
This allows the network to represent not just the expected value of actions, but also the confidence in those estimates. Combining these, we arrive at the following equation for the action signal v., (Q):
A
Q(5,a)
(a,, 0r2 (a))²
T. (Q(s, a)) = a=1 V2TO
exp
(
20
This equation represents the input to each neuron as sum of Gaussian functions, where each function centered on an action's 's direction and scaled by its Q-value.
Full excitatory neuron dynamics. The dynamics of the excitatory neurons in the ring attractor, now incorporating the Q-value inputs, are described by:
du, 1 ( dt max
M==N
0.
Σ m=1
w
Wmn Um + (Z) + (T-> F.)) 2
(E++E)
(6)
This equation captures how the activation of each neuron evolves over time, influenced by the Gaussian
functions of the input action value z,,(Q), the excitatory feedback en > w Fm F.,) Ums and the inhibitory feedback 7/, = =w((-=..)
Full inhibitory neuron dynamics. The complete dynamics of the inhibitory neuron are given by: du 7
( maar 0.
(0. 2
N
Un 1) u,
w EE,
5

---

where the self-inhibition term u and excitatory input Up from each excitatory neuron from Eq. 6 updates the activation of the inhibitory neuron de
du Eq. (6) and Eq. (7) collectivcly de-
scribe the dynamics of the ring attractor network, capturing the interplay between excitatory and inhibitory neurons, external inputs, and synaptic connections. To translate the ring attractor's 's out- put into an action in the 2D space, we use the following equation:
P4 dg
N2
Ue
No
((4) action argmax V N(E)
Excitatory connections (V.))
Inhibitory connectins (UA)
4a 01
Overall ectivation
where n fI...., N(E), (EE is the number of excitatory neurons the ring attractor, ((4)
Excitatory neuron
((..))
Inhibitory neuron
Attractor
is the number of discrete actions in the action space A, and the activation of excitatory neurons
around the ring V 11, 02. [.(m]] This equation assumes that both the newrons in the ring attractor and the actions in the action space are uniformly distributed. This approach allows for nuanced action selection that takes into ac- count both the spatial relations between actions and their estimated values. visualization of the ring is presented Fig.
Figure I: Ring attractor Touretzky representation: Circular arrangement of excitatory neurons (NO- N7) and central inhibitory neuron. Four input sig- nals shown as colored gradients. Overall activa- tion depicted by red outline. Includes connection weights and input signal parameters, illustrating the final ring attractor dynamics state from Eq. (8)
3.1.3 Uncertainty Quantification Integration
In the field of DRI,, for any state-action pair (s, a), the Q-value (ss, a) can be expressed as function of the input state through function approximation algorithm PP(s) taking input the current state s. This function approximation algorithm (#g(s) can be expressed as the weight matrix of our function approximation algorithm transposed 6 times the feature vector extracted from the input state zss): (s..a) a)
(s)s)) HT (s) 34 For further context on function approximation in RL see Appendix A.2 As stated in Section 3.1.2 the variance of the Gaussian functions, input the ring attractor, will be given by the variance of the estimate value for that particular action O; = og. Among the various methods to compute the uncertainty of the action (a.)) have chosen compute a posterior distribution with Bayesian linear regression (BLR). BLR acts as output layer for our neural network (NN) of choice.
Algorithm 1 CTRNN Ring Attractor Action Selection
We choose linear regression model because does not compromise the ef- ficiency of the NN, while at the same time it provides distribution com- pute the variance for the state-action pairs. The implementation is based on [1, where a Bayesian value-based DQN model was instantiated output an uncertainty-awar prediction for the state-action pairs. The new Q function defined as:
Require: Input: State 8, action from Action space
A.
1: Step 1: Compute Q-values and Uncertainty
2: Compute mean Q(8, a) and variance of using Eq.
11
3: Step 2: Generate Input Action Signals
4: for each excitatory neuron n do
5: Generate Gaussian action signal using Eq. 5
6: end for
7: Step 3: Reach Attractor State
8: for timestep until T do
D choose 7=50
(s, (0()) was
empirically
9:
for each excitatory neuron do du., Update excitatory neurons using Eq. 6
where Wr are the weights from the pos- terior distribution of the BLR model. Eq. (9) represents the parameters of the final Bayesian linear layer. When provided with state transition tuple (s, a, r,),,'). s'). where s is the current state, a is the ac- tion taken, is the reward received, and is the next state. This tuple represents single step of interaction between agent and environment. The model learns to adjust the weights
10:
11:
end for
12:
du Update inhibitory neuron using Eq. 7
13: end for
14: Step 4: Translate Neural Activity V to action se-
lected from action space A using Eq. 8
15: return action

---

Wa of the BLR and the function approximation algorithm, i.e. neural networks (NNs) (Φg), to align the Q values with the optimal action a argmar((;90 Wa).
Q(s,a) ( + y : + Donge (8') Wnireet
(10)
where y is the discount factor, y is the expected Q-value, 4 Doug. are the features from the next state s' extracted by the function approximation algorithm & using the target network parameters, Otarget refers to the target function approximation algorithm parameters, and â is the predicted optimal action in the next state s'. The construction of the Gaussian BLR prior distribution and the weights i sample aa collected from the posterior distribution are performed through Thompson Sampling. This process allows us incorporate uncertainty into our action-value estimates. For details on the construction of the Gaussian prior distribution and the specifics of the sampling process, we refer the reader to []] Both the mean Q(s, a) and the variance 0º of Eq. 11 are calculated from finite number of samples I.
((s,0) Eiso (is,))=Ei0((s.. as)
Ei wa 0o(s)
Eiso (w",, Pa(()) ma)²
G2
(11)
I
3.2 Deep Learning Ring Attractor Model
To further enhance the ring attractor's integration into RL frameworks and agents, we provide Deep Learning (DL) implementation. This approach improves model learning and integration with DRL agents. Our implementation offers both algorithmic improvements, by benefiting from DL training process, and software integration improvements, easing the deployment processes. Recurrent Neural Networks (RNNs) offer an approach for integrating ring attractors within DRL agents. Recent studies by 2 show that RNNs perform well in modeling sequential data and in capturing temporal dependencies for decision-making, Like CTRNNs, RNNs mirror ring attractors' temporal dynamics, with their recurrent connections and flexible architecture emulating their interconnected nature. Allowing modelling weighted connections for forward and recurrent hidden states, as shown in Appendix A.6 The premises for modeling RNN are as follows.
Attractor state recurrent connections. RNN recurrent connections model the attractor state, integrating information from previous time steps into the current network state, allowing for the retention of information over time.
Signal input as forward pass. Forward connections from previous layers are arranged circularly, mimicking the ring's 's spatial distribution. The attractor state encodes the task context, influenced by the current input and hidden state. A learnable time constant т, inherited from Eq. (2) controls input and temporal evolution, enabling adaptive behavior and elastic contribution to the attractor state.
3.2.1 DRL Agent Integration
To shape circular connectivity within RNN, the weighted connections in the input sig- nal V(s) and the hidden state or attractor state U(v) are computed as in [ This circu- lar structure mimics the arrangement of excitatory neurons in the ring attractor. Eq. (12) shows the input signal to the recurrent layer Vm,= from neuron m from the previous layer in the DL agent to neuron in the RNN. The hidden state, Um.. mimics an attractor state, representing the recurrent connections in the RNN. The weighted RNN connections include fixed input-to-hidden connections (w m, n) to maintain the ring's spatial structure, and learnable hidden-to-hiddes connections "= m, n) to capture emerging action relationships. d(m.n)
These depend on a parameter that drives the decay over distance and distance between neurons d(m, n) where N is the total number of neurons for the RNN and M is the count of neurons in the previous layer of the NN architecture. The function do (8) S - A
V(s)m, = (s)
1 mmnn
M
M
d(m,n) = min (Im - n
N |mm n
N
N
U(v)m,a = h((v) " H H = h h( Te d(mn.)
maps the input state of the DL. agent to a representation of characteristies that will be
d(m, n) = min (Im - nl. N - Im nl)
(12)
the input of the recurrent layer. Likewise, 0 represents the parameters of this function (i.e., the weights and biases of the NN layers preceding the RNN layer, which extract relevant features from the input). The function h((v) R - R is parameterized by learnable weights that map the information from previous forward passes to
7

---

the current hidden state. The learnable parameter T is the positive time constant responsible for the integration of signals in the ring. Here, acts as learnable scaling factor that controls the balance between immediate input responsiveness and temporal integration within the ring attractor dynamics, with smaller values increasing sensitivity to current imputs and enabling faster adaptation, while larger values promote stability and smoother transitions between attractor states by integrating information over longer time horizons. It defines the contribution of input do (s) to the current hidden state, imitating the attractor state, applied to NNs.
Finally, the vetion-value function Q(s, a) is derived from the RNN layer's output by applying the neurons activation function to the combined input V(s) and hidden state information Us). The activation function of choice is hyperbolic tangent tanh, this function is symmetric around zero, leading to faster convergence and stability. However, the output range of tanh (-1 to 1) is not fully compatible with value-based methods, where the DL agents need output action-value pairs in the range of the environment's 's reward function. To address this issue and prevent saturation of the tanh activation function,
scale the action-value pairs by multiplying them by learnable scalar B, as (u)""" Q(s, a) 8hg = 3 tanh (V(s) + U())) = 3 tanh (2 (s:) " f- h h """"" "H"). While our main focus on value-based ring-attractor agents, we also provide an overview of methods and results for policy-gradient approaches in Appendix A.3
4 Experiments
This section presents the findings of our experiments that validate our proposed approach to inte- grate ring attractirs into RL algorithms. To assess the effectiveness of our method, we conducted comparisons between multiple baseline models and action spaces. The evaluation encompasses two implementations: a CTRNN exogenous ring attractor and a DL approach where the ring attractor is modeled directly into DRL agents. In both implementations, the action-value pairs Q(s a) are evenly distributed across the ring circumference. For the exogenous model, each action associated with a specific discrete angle or continuous space, section 3.1.2] In the DL implementation, each neuron in the RNN corresponds one action-value. The ring attractor serves the output layer of the DL agent with the weights modeling the circular topology of the action space, Section For both approaches, ring attractor agents annotated with the suffix RA. We demonstrate that ring attractors significantly enhance action selection and speed up the learning process.
Super
Learning
Openal Highter
0006 STD
Figure 2: Learning speed comparison. Above: OpenAI Gym Super Mario Bros environment 15 with discrete action space. Below: OpenAl highway (21 with a continuous 1-D circular variable. The plot shows cumulative reward over 1 million frames for three models: Standard BDQN; BDQNRA with ring attractor behavior policy from Section 3.2. setting the action-value pair variance constant to o. E. using this fixed variance to enable smooth action transitions while preventing interference with opposing actions; and BDQNRA-UA with RA and Uncertainty Awareness (UA) implementing the uncertainty quantification model from Section 3.1.3 to feed into the variance of the action-value pairs. Displaying mean episodic returns over 10 averaged seeds.
4.1 CTRNN Ring Attractor Model Performance
To evaluate our CTRNN ring attractor model from Section 3.1.2 integrated with BDQN || performed experiments in the OpenAl Super Mario Bros environment [5] and OpenAl highway 21 These benchmarks exhibit a spatially distributed complex decision-making scenario with 8 discrete different actions and navigation- iwwigatiooncentric task with a continuous 1D cireular actions space, respectively. Fig. shows that both ring attractor models (BDQNRA A and BDQNRA-
consistently outperform standard BDQN. The uncertainty-aware version (BDQNRA-UA) shows the best overall performance, highlighting the benefits of combining ring attractors spatial distribution of the action space with uncertainty- aware action selection. Empirical evaluations revealed that WIRNN-bbase ring attractor
8

---

models exhibited mean computational overhead of 297.3% (SD 14.2%) compared to the baseline, significantly impacting runtime. To address this performance bottleneck and integrate the spatial understanding of the ring attractor into DRL, we developed a DL implementation of the ring attractor. This DL implementation is evaluated in the following subsections.
4.2 DL Ring Attractor Model Performance
To evaluate the effectiveness of our DL-bbased ring attractor implementation from Section 3.2 we conducted experiments implementing DDQNRA, an extension of DDQN 43) Fig. 3 shows DDQNRA consistently outperforms standard DDQN, with the ring attractor's spatial encoding ability contributing significantly to improved learning speed. These results demonstrate substantial performance gains, especially in environments with strong spatial and directional cues.
Learning comparteor:
Maro
Learning comparison: OpenAA
DOON ETD
Geootedd
1221
E00N
Figure 3: Performance comparison: DDQNRA vs standard DDQN
in two environments. Above:
OpenAI Super Mario Bros [15). demonstrating adaptability to complex, game-like scenarios. Below: OpenAl highway (21), showing learning speed in spatial navigation tasks. Displaying mean episodic
returns over 10 averaged seeds.
4.3 Performance on Atari 100k Benchmark
In this results section, we provide comprehensive analysis of the performance of our DRL model on the Atari 100k benchmark (5). We present comparisons with state-of-the-art models, highlighting the improvements achieved by our approach. Table presents a comparison of our ring attractor- based
DRL model integrated with Efficient Zero 488 which evaluates performance in Atari games with
limited training size of 100,000 environment steps. The table includes results from top perperforming algorithms SPR [30] and CURL. 332 for context. Our model demonstrates significant improvements over baseline methods, achieving 53% average improvement, with over 100% gains in games with inherent spatial components, like Asterix and Boxing. Results demonstrate ring attractors provide consistent performance improvements across diverse RL tasks, with benefits extending beyond purely spatial tasks. We re-run baseline EffZero under identical cloud resources, to ensure fair comparison.
The mapping between game action spaces and ring configurations reflects the fundamental structure of each environment's action space. Games with primarily directional movement actions, such as Asterix and Ms Pacman, utilise a Single ring configuration where eight directional movements map naturally to positions around the ring circumference. In contrast, games combining movement with independent action dimensions, such as Seaquest and BautleZone, employ a Double ring configuration, one ring encoding movement actions and another representing secondary mechanics such as combat. This architecture maintains spatial relationships while preserving the independence of different action types. Further implementation details for multiple ring dynamics can be found in Appendix A.7
Ablation studies were conducted to isolate the impact of key components in our ring attractor models, detailed in Appendix A..4.] and A.4.2 for both implementations respectively. For the exogenous model, we compared performance with correct and randomized action distributions in the ring. In the DI. implementation, Additionally, Appendix [e.5]r preserved attractor dynamics behaving as a low-pass filter the CTRNN and DL both at pre and post-training stages.
removed the circular weight distribution to assess its importance.

---

Table 1: Performance comparison on Atari 100k Benchmark [5]. Benchmark performed across all environments where actions can be laid one more 2D action space planes (ring attractors). This is represented by the ring configuration column. Game score and overall mean and median nuraa-normalized scores recorded the end of training and averaged over 10 random seeds, samples per seed.
Game
Agent Human
Reported SPR
Implemented
Environment
Ring
CURL
EffZero
EffZero
EfZzrrRA
Alien Asterix
Double
7127.7
558.2
801.5
808.5
738.1
1098.8
Single
8503.3
734.5
977.8
25557.8
14839.3
31037.3
Bank Heist
Double
753.1
131.6
380.9
351.0
362.8
460.5
BattleZone
Double
37187.5
14870.0
16651.0
13871.2
11908.7
15672.0
Boxing
Double
12.1
1.2
35.8
52.7
30.5
62.4
Chopper C.
Double
7387.8
1058.5
974.8
1117.3
1162.4
1963.0
Crazy Climber
Single
35829.4
12146.5
42923.6
83940.2
83883.0
100649.7
Freeway
Double
29.6
26.7
24.4
21.8
22.7
31.3
Frostbite
Double
4334.7
1181.3
1821.5
296.3
287.5
354.8
Gopher
Double
2412.5
669.3
715.2
3260.3
2975.3
3804.0
Hero
Double
30826.4
6279.3
7019.2
9315.9
9966.4
11976.1
Jamesbond
Double
302.8
471.0
365.4
517.0
350.1
416,4
Kangaroo
Double
3035.0
872.5
3276.4
724.1
689.2
1368.8
Krull
Double
2665.5
4229.6
3688.9
5663.3
6128.3
9282.1
Kung Fu M.
Double
22736.3
14307.8
13192.7
30944.8
27445.6
49697.7
Ms Pacman
Single
6951.6
1465.5
1313.2
1281.2
1166.2
2028.0
Private Eye
Double
69571.3
218.4
124.0
96.7
94.3
155.8
Road Runner
Double
7845.0
5661.0
669.1
17751.3
19203.1
29389.3
Seaquest
Double
42054.7
384.5
583.1
1100.2
1154.7
1532.8
Human-normallsed Score
Mean
1.000
0.428
0.638
1.101
0.959
1.454
Median
1.000
0.242
0.434
0.420
0.403
0.531
Conclusion and Future Work
Conclusion. This paper presents novel approach RL, integrating ring attractors into action selection. Our work demonstrates that these neuroseience-inspired ring attractors significantly enhance learning capabilities for RI. agents, leading more stable and efficient action selection, particularly in spatially structured tasks.
The integration of ring attractors DL module proves to be particularly effective, allowing for end- to-end training and easy incorporation into existing RL architectures. Our experiments demonstrate significant improvements in action selection and learning speed, achieving state-of-the-art results on the Atari 100K benchmark. Notably, observe average 53% performance increase across all tested games compared to prior baselines. The most relevant improvements were observed in games with strong spatial components, such as Asterix (110 % improvement) and Boxing (105% % improvement). Additionally, we observed improvements in other environments tested outside the Atari benchmark, further supporting the effectiveness of our approach across RL tasks and agents.
Future work. Future research should investigate scalability in high-dimensional action spaces and explore its efficacy in domains where spatial relationships are less straightforward. We believe that the success of this approach opens up several future research paths. This work can be extended to multi-agent scenarios and continuous control tasks as these have been the current limits of this piece of research. The ring attractor approach may exhibit reduced applicability discrete action spaces with cardinality |A| < 3 where the circular topology overhead exceeds potential spatial encoding benefits, and in action spaces where the action correlation matrix approaches the identity matrix, indicating absence spatial dependencies or sequential relationships between discrete actions. Additionally, extension to 3D spatial navigation tasks presents an organic expansion of this work, these scenarios will naturally introduce realistic sensor noise into the system.
the field of incertainty-awan decision making, leveraging the spatial structure provided by attractor networks presents promising avenue to map uncertainty explicitly to the action space. Deploying the techniques presented here into specific domains could yield performance boosts, especially in safe RI., leveraging their stability properties to enforce constraints and ensure predictable behavior. This approach not only improves performance but also offers potential insight into spatial encoding of actions and decision-making processes, bridging the gap between neuroscience insseee-dierdddd models and practical RL agents.
10

---

Acknowledgments and Disclosure of Funding
This work was conducted part of a self-funded PhD programme the University of Manchester. Computational resources were provided by the Computational Shared Facility 3 (CSF3) at the University of Manchester [1]]. We gratefully acknowledge the support of the Research IT team for maintaining and providing access to the GPÚ Cluster.
Funding: This research received no external funding.
Competing Interests: No competing interests are declared.
References
[1] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep n-networks. CoRR, abs/ 2018.
[2] Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy P. Lillicrap. Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayíl, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil C. Rabinowitz, Razvan Pascanu, Charlie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid-like representations artificial agents. Nat., 557(7705):429 433,22994333-2 2018.
[3] Victor Bapst, Alvaro Sanzaez-Gonzalez, Carl Doersch, Kimberly L. Stachenfeld, Pushmeet Kohli, Peter W. Battaglia, and Jessica B. Hamrick. Structured agents for physical construction. CoRR, abs/1904.03177 2019.
[4] Randall D. Beer. On the dynamics of small continuous- time recurrent neural networks. ISAB, htw/1599.066611 1995.
[5] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. CoRR, abs/1 204.41207.708.. 2012.
[6] Richard E. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957.
[7] Hugh Blair, Di Wu, and Jason Cong. Oscillatory neurocomputing with ring attractors: A network architecture for mapping locations in space onto patterns of neural synchrony. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 369:20120526, 02 2014.
[8] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. CoRR, abs/18 10.944 2018.
[9] Eugene Bykovets, Yannick Metz. Mennatallah El-Assady, Daniel A. Keim, and Joachim M. Buhmann. How to enable uncertainty estimation in proximal policy optimization, 2022.
[10] Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks perform spatial localization, 2018.
[11] Nikita Durasov, Timur M. Bagautdinov. Pierre Baqué, and Pascal Fua. Masksembles for uncertainty estimation. CoRR, abs/2012.0883:3 2020.
[12] JP Goodridge, PA Dudchenko, KA Worboys, EJ Golob, and Taube IS. Cue control and head direction cells. The Journal of neuroscience: the official journal of the Society for Neuroscience, 112:RC154, 08 1998.
[13] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni- tive mapping and planning for visual navigation. CoRR, abs/1 702.0990,, 2017.
[14] Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. DEUP: direct epistemic uncertainty prediction. CoRR, abs/2102.08500 2021.
[15] Christian Kauten. Super Mario Bros for OpenAI Gym. GitHub, 2018.

---

[16] Mikail Khona and Ila R Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12);744-76, December 2022.
[17] Zachary P. Kilpatrick and Bard Ermentrout. Wandering bumps in stochastic neural fields. SIAM Journal Applied Dynamical Systems, 12(1):61-94, 2013.
[18] Sung Soo Kim, Hervé Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dynamics in the drosophila central brain. Science, 35666340):849-833,, 2017.
[19] Anna Kutschireiter, Melanie A. Basnak, Rachel I. Wilson, and Jan Drugowitsch. Bayesian inference in ring attractor networks. Proceedings of the National Academy of Sciences, I 120(9):a22102211 2023.
[20] Hyun-Suk Lee, Yao Zhang, William Zame, Cong Shen, Jang- Won Lee, and Mihacla van der Schaar. Robust recursive partitioning for heterogeneous treatment effects with uncertainty quantification, 2020.
[21] Edouard Leurent. An environment for autonomous driving decision-making 2018.
[22] Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, and Ji He. Recurrent reinforcement learning: A hybrid approach, 2015.
[23] BL McNaughton, CA Barnes, JL Gerrard, Gothard, MW Jung, JJ Knierim, H Kudrimoti, Y Qin. WE Skaggs, M Suster, and KL Weaver. Deciphering the hippocampal polyglot: the hippocampus as path integration system. Exp Biol., 12, 1996.
[24] Hasselmo ME. Grid cell mechanisms and function: contributions of entorhinal persistent spiking and phase resetting. Hiipoocampus, page 19021258, 12 2008.
[25] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments, 2017.
[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):5295333 2015.
[27] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. CoRR, 100..6602., 2016.
[28] Jesús D. Rivero-Ortega, Juan S. Mosquera- -atturaa,, Josh Pardo-Cabrera, Julián Hurtado- López, Juan D. Hernández, Victor Romero-Cano, and David Ramírez-Moreno. Ring attractor bio-inspired neural network for social robot navigation. Frontiers in Neurorobotics, 17, 2023.
[29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[30] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR, abs/2007.05929, 2020.
[31] Satpreet H Singh, Floris van Breugel, Rajesh P. N. Rao, and Bingni W. Brunton. Emergent behaviour and neural dynamics in artificial agents tracking odour plumes. Nature, 2023.
[32] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: contrastive unsupervised repre- sentations for reinforcement learning. CoRR, abs/2004.04 1431, 2020.
[33] Xuelong Sun, Shigang Yue, and Michael Mangan. A decentralised neural model explaining optimal integration of navigational strategies in insects. eLife, 9:c54026, jun 2020.
[34] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, 2018.
12

---

[35] Richard S. Sutton, Hamid R. Maei, and Csaba Szepesvári. A convergent o(n) temporal- difference algorithm for pfllcy learning with linear function approximation. Advances in Neural Information Processing Systems (NewrIPS), 21:1609-1616, 2009.
[36] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999.
[37] Jeffrey Taube. The head direction signal: Origins and sensory-motor integration. Annual review of neuroscience, 30:181-207, 02 2007.
[38] JS Taube. Head direction cells recorded in the anterior thalamic nuclei of freely moving rats. Journal of Neuroscience, 15(1):70-86, 1995.
[39] Emanuel Todorov. Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 0026-5033. IEEE, 2012.
[40] David Touretzky. Attractor network models of head direction cells. The Journal of neuroscience the official journal of the Society for Neuroscience, pages 411-432, 07 2005.
[41] University of Manchester. Computational Shared Facility 3 (CSF3).
https://ri
Stservices.maanee www.meencceee...mmmmmmmm. ac. ac.uk//sf3// 2024.
[42] Pantelis Vafidis, David Owald, Tiziano Albis, and Richard Kempter. Learning accurate path integration in ring attractor models of the head direction system. eLife, 11:669411, jun 2022.
[43] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. CoRR, htss/1509.06661, 2015.
[44] Jane X Wang. Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-rriiforrcement learning system. Nature neuroscience, (1(6))860 868, 2018.
[45] Xiao-Jing Wang. Probabilistic decision making by slow reverberation in cortical circuits. Neuron, 36(5):955-968, 2002.
[46] Rachel I. Wilson. Neural networks for navigation: From connections to computations. Annual Review of Neuroscience, 46(Volume 46, 2023):403-423, 2023.
[47] Yirong Xiong and Vivek Hari Sridhar. Understanding the influence of uncertainty and noise on spatial decision dynamics. Cognitive Computational Neuroscience, 2024.
[48] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. CoRR, abs/211 100220, 2021.
[49] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019.
[50] Kechen Zhang. Representation of spatial orientation by the intrinsic dynamics of the head- direction cell ensemble: a theory. Journal of Neuroscience, (6(6):2112-24) 1222166 1996.
[51] Michaël Zugaro. Alain Berthoz, and Sidney Wiener. Background, but not foreground, spatial cues are taken references for head direction responses by rat anterodorsal thalamus neurons. The Journal of neuroscience: the official journal of the Society for Neuroscience, 21:RC154, 08 2001.
13

---

A Appendix
A.1 Background: Attractor Networks Theoretical Foundations
Ring attractor networks are type of biological neural structure that has been proposed underlie the representation of various cognitive functions, including spatial navigation, working memory, and decision-makinn [88
Biological intuition. In the early 1990s, the research carried out by [50] proposed that ring neural structures could underlie the representation of heading direction in rodents. (50) argued that the neural
activity in an attractor network might encode the direction of the animal's head, with the network transitioning from one attractor state to another state as the animal turns.
Empirical evidence. There is growing evidence from neuroscience supporting the role of ring attractors in neural processing. For example, electrophysiological recordings from head direction cells (HDCss) of rodents have revealed a circular organization of these neurons, with neighbouring HDCs encoding slightly different heading directions (38) Furthermore, studies have shown that HDC activity can be influenced by sensory inputs, such as visual signals and vestibular signals, and that these inputs can cause the network to update its representation of heading direction [7 47 showed model internal noise biases toward accuracy over speed, while environmental uncertainty exhibits a U-shaped effect, where moderate uncertainty favors speed and extreme uncertainty favors accuracy. [46] demonstrated that biological navigation networks rely attractor dynamics while maintaining adaptability through continuous synaptic plasticity and sensory feedback.
Sensor fusion in ring attractors. Ring attractor networks provide a theoretical foundation for under- standing cognitive functions such as spatial navigation, working memory, and decision decision-making. In the context of action selection in RL, sensor fusion plays a pivotal role in augmenting the information- processing capabilitiess of these networks. By combining data from various sensory modalities, ring attractors create more nuanced and robust representation of the environment, essential for adaptive behaviors (24). Research has elucidated the relationship between ring attractors and sensory inputs, with the circular organisation of HDCs in rodents complemented by the convergence of visual and vestibular inputs, highlighting the integrative nature of sensory information within the ring attractor framework [5]].
Modulation by sensory inputs. Beyond the spatial domain, sensory input dynamically influences the activity of ring attractor networks. Studies have shown that visual cues and vestibular signals not only update the representation of heading direction but also contribute the stability of attractor states, allowing robust spatial memory and navigation [2]
Sensor fusion for action selection. The concept of sensor fusion within the context of ring attractors extends beyond traditional sensory modalities, encompassing diverse sources, such as proprioceptive and contextual cues [23]. Building on the foundation of ring attractor networks discussed earlier, the integration of sensor fusion in the context of action selection involves fusing the action values associated with each potential action within the ring attractor framework. In particular, sensory information, previously shown to modulate the activity of ring attractor networks, extends its influence the representation of action values. The inclusion of sensory information reflects higher cognitive process, where the adaptable nature of ring attractor networks supports optimal decision makinng and action selection in complex environments.
A.2
Function Approximation for Q- Q-Values
A.2.1
Motivation and Background
In RL. an agent optimizes behaviour by interacting with an environment modeled as Markov Decision Process (MDP), .M- = AS 7) where S is the state space, A is the action space, S x A x S + [0, 1] is the state transition probability function, R S x A x S -+ R is the reward function, and y E [0, 1) denotes the discount factor. The agent seeks to learn policy 'T s). which defines the probability of selecting action a in state 8, that maximizes the expected discounted
return 34
oc Σ 2'
( E,
(13)
where J(T) is the objective function representing the expected total return (performance measure) of policy n. (so, do, To, $1, 01, ... represents trajectory of states, actions, and rewards, Tt
14

---

is the reward received at time step t, is the discount factor that balances immediate versus future rewards, and the expectation is taken over trajectories sampled by following policy я.
Action-value functions are foundational to various value-based RI. algorithms. The action-value function is defined as:
Q"(s,a)= E, Σ r' 80
do
a
(14)
which quantifies the expected cumulative discounted reward when executing action in state and subsequently following policy и for all future time steps.
This function satisfies a recursive relationship known as the Bellman equation 6
(=) - - P.P P,R r Ex(d' 80)
(15)
where Ey', PR denotes expectation with respect the next state s' and reward r following the environment dynamics defined by P and R, and r + 7 π (a' s')Q" a') is the expected immediate reward plus the discounted value of future actions. This forms the mathematical basis for value iteration, policy iteration, Q-lcarning, and other dynamic programming and temporal-dine -ifferrene methods used to compute approximate Q*
In environments with large or continuous state-action spaces, storing tabular representations of Q(s, a) becomes infeasible due to memory constraints and the inability to visit all state-action pairs sufficiently often. Even in moderately sized discrete domains, function approximation is often employed to leverage generalization across similar states or actions, thereby speeding up learning and reducing data requirements. By using function approximators, RL algorithms can handle high-dimensional inputs and share learned structure across different parts of the state-action space.
A.2.2 General Formulation
Q-functions are approximated using parameterized functions Q(s, a; 0), where Rd are learnable weights:
(16) where the target Q' could be the Q-function of the current policy (policy evaluation) or the optimal Q-funetion Q' (control). The goal is learn such that Q(s, a; 0) closely estimates the target Q-values across the relevant state-action space.
Q(s, 6) Q" (s, a).
Parameters 0 are typically updated through stochastic gradient descent to minimize a loss function. For instance, using the mean squared Bellman error:
C(8) Essa,r.a'' [(r max ((''''':''''' Q(s, a;)))
(17)
where 6' represents parameters of a target network that is periodically updated to match 0 (in deep Q- learning algorithms such as DQN (26), or 0' = 0 in simpler settings like traditional Q-learning). The term r + max,' Q(s', a'; 0') is often called the "target" value, while Q(8, a; 0) the "prediction". The parameters are updated according to:
0c-1 -.1=0--a
(18)
where is the learning rate, and VoL(O:) is the gradient of the loss function with respect to 8,. In practice, these updates are performed using stochastic gradient descent or variants on mini-batches of experience to improve computational efficiency and stability.
A.2.3 Linear Function Approximation
straightforward approach linear function approximation, where state-action pairs are mapped into a feature space via feature function Φ S x A -> RRd, and Q-values are computed as:
Q a; 0) 0 a))
(19)
This formulation is computationally efficient and, under specific conditions, enjoys theoretical convergence guarantees in on-policy settings (35]. However, linear approximation's 's representational
15

---

capacity is fundamentally constrained by the chosen feature representation o(s, a). For discrete action spaces, an alternative formulation represents a separate linear function for each action:
a;) 0g ((s),
(20)
where (ss) E Rd state feature vector and 0g R is parameter vector specific action
The transition from linear to neural approximation typically retains this foundational structure while enhancing the feature extraction process. Instead of relying on handcrafted features, neural networks learn feature representations automatically through their hidden layers.
A.2.4 Neural Function Approximators
To capture richer, nonlinear dependencies, neural networks can be used parameterize Q-functions. In the field of DRL, for any state-action pair (s, a), the Q-value Q(s, a) can be expressed as function of the input state through a function approximation algorithm @g(s). This notation evolves from the linear case but generalizes allow for arbitrary nonlinear transformations of the input. Formally:
Q(s = (5(s)) 0 x((),
(21)
where rr(s corresponds the (learedd) feature representation of the state (e.g., the outputs of the penultimate layer), and 0 represents the weights of the final layer.
With this design, neural networks learn both the feature extraction (r(()) and the final mapping (0) end-to-end through backpropagation, enabling highly expressive function approximators that can generalize across large or complex state-action domains.
A.3 Policy Gradient Methods with Ring Attractors
While previous sections focused on value-based methods, ring attractors can also enhance policy gradient methods, such as Proximal Policy Optimization (PPO) (29]. The integration follows similar principles, but adapts to the unique characteristics of policy-based approaches [36].
A.3.1 Proximal Policy Optimization with Ring Attractors
In standard PPO, the policy is directly parameterized by neural network that outputs action probabilities. When integrating ring attractors, we modify this structure to incorporate the spatial relationships between actions through the ring topology.
Ring Attractor Policy Network. For PPO integration, we adapt the DL ring attractor structure from Section 3.2 While the value-based methods use the network to process Q-values, in PPO we process policy logits. For policy network with parameters the initial policy logits are computed as:
Z(s) = 4s(s))
(22)
This parallels Section 3.2 where Q(s, a) based methods. The difference is that Z(s) represents policy logits rather than action- values.
(s)s)) represents the output of the network for value-
These logits are then processed through the ring attractor using the same weighted connection structure. The input signal ((s) uses 7(s) input and follows the same form in Eq. (12
J-H- = 2() dimnn V(.) = Zs)) w m.
(23)
Similarly, the hidden state Ulv) maintains the same recurrent dynamics as in Eq. 12 difference occurs in the final output layer, where instead of scaling the output to represent Qavaues, we apply softmax function to directly obtain action probabilities:
The key
ग (aa s) softmax(tanh(((()) (()))))
(24)
16

---

A.3.2 Policy Gradient Experimental Results
As shown in Figure PPO 29 with ring attractors (PPO-RA) demonstrates significant performance improvements over standard PPO in the OpenAl Gym Super Mario Bros environment [[]
Learning speed comparison: OpenAl Gym
Super Bros
200
PPO
PPO-RA PPO-RA STD
PPO Wilrrommee
PPO STD
171.0
Mosinn Aug Rowwerd
Frames
Figure 4: Performance comparison: PPO-RA vs standard PPO in OpenAl Gym Super Mario Bros environment (15] with discrete action space, demonstrating adaptability to complex, game-like scenarios. The plot shows PPO-RA (red) consistently outperforming standard PPO (blue), with shaded regions representing standard deviation across 10 averaged seeds.
Our experimental results reveal an average performance increase of +71.4% over standard PPO, with final performance improvement of +67.5% and a maximum performance gain of +16..4% during training. These results demonstrate that the spatial awareness provided by ring attractors is beneficial across different RL paradigms, just value-based methods. The performance gains in PPO-RA can be attributed to more efficient policy updates due to the preservation spatial relationships between actions, smoother policy gradients resulting from the structured representation of the action space, and enhanced exploration through the natural diffusion of policy preferences across similar actions in the ring.
PPO-RA converges faster and achieves higher rewards than standard PPO. This suggests that ring attractors provide a general enhancement mechanism applicable to multiple RI. approaches, show- casing that their utility may extend beyond the value-based methods explored in previous sections.
A.4 Validating Ring Attractor Contributions Through Ablation Studies
A.4.1 Exogenous Ring Attractor Model Ablation Study
To isolate the impact of the ring attractor structure, we conducted an ablation study comparing our full BDQNRA model against versions with the action space overlay in an incorrect distribution in the ring. Fig. 5 This incorrect distribution involves randomly rearranging the placement of actions within the ring, disrupting the natural topology of the action space.
For instance, this could mean placing opposing or unrelated actions side by side in the ring, such as pairing "move left" with "move down" instead of its natural opposite "move right". More generally, this incorrect distribution breaks the inherent relationships between actions that are typically preserved in the ring structure.
17

---

Ablation Study: OpenAl Gym Super Marto Bros
BBOM B9th
D0CINTA-TM DOONTSG-EE
BOGNI16-UA 511
2000
Frames
Figure 5: Ablation study comparing BDQN variants in OpenAl Gym Super Mario Bros 15]. The plot shows cumulative reward over I million frames for three models: Standard BDQN [I: BDQNRA-UA with RA and Uncertainty Awareness (UA) implementing both the ring attractor behavior policy from Section 3.1.2 and the uncertainty quantification model from Section 3.1.3 and BDQNRA- applying the same concepts from BDQNRA-UA, but randomly distributing the action space across the ring in each experiment. Displaying mean episodic returns over 10 averaged seeds.
A.4.2 Deep Learning Ring Attractor Model Ablation Study
This ablation study focused isolating the impact of the ring-shaped connectivity in RNN-based ring attractor model. The key aspect of our experiment was to remove circular weight distribution in both the forward pass (input-to-hidden connections) and the recurrent connections (hidden-to- hidden). while maintaining all other aspects of the RNN architecture. This approach allows to directly assess the contribution of the spatial ring structure to the model's 's performance.
Ablation study, AWN ring atoector ingi envertttiion Atari LOOK RacMan
Atradion study: HWN ring attractor implemertation on Atpri 100K Coopper Command
D'GariiA
$1000
DPCaee
4tags
Figure 6: Ablation study results comparing the performance of the full RNN-based ring attractor model against a version with the circular weight distribution removed. 'he graph illustrates a significant performance drop for the Ms Pacman and Chopper Command environments in the Atari 100K benchmark [5). This emphasizes the role of the circular topology encoding spatial information and enhancing learning. Displaying mean episodic returns over 10 averaged seeds.
In our original model. the weights between neurons were determined by distance-der -depended function that created a circular topology. This function assigned stronger connections between neurons that close together in the ring and weaker connections between distant neurons. For the ablation, we replaced this distance-dependent weight function with standard weight matrices for both the input-to-hidden and hidden-to-hided connections. This modification effectively transforms ring
18

---

attractor RNN into a standard RNN, where the weights are not constrained by the circular topology. We retained other key elements of the model, such as the learnable time constant and the non-linear transformation, to isolate the effect of the ring structure specifically.
A.4.3 Deep Learning Ring Attractor Model Evolution
In this appendix section, we analyze model dynamics with both forward pass (V(s)) and hidden- to-hidden (U() weights made trainable, rather than the standard approach of fixed forward pass connections, as presented in Section 3.2 As shown in Fig. 7 the forward-pass connections preserve the ring structure over training time, with strong distance-dependen decay patterns maintained throughout the learning process. This may indicate that the network naturally favors maintaining spatial topology for transmitting sensory information on a per-frame basis.
Neuron Weights in RNN Forward Pass (Input Hidden)
Step:
Step: 300000
Step: 600000
Step: 1000000
Figure 7: Evolution of forward pass weights showing preserved distance-dependent decay over training time, maintaining ring structure for spatial information transmission.
The hidden-to-hidder connections. depicted Fig. 8 exhibit markedly different behavior, These connections evolve beyond their initial ring structure, developing specialized patterns that enable the relationships between neurons in the hidden space. This flexibility
encoding of environment-speccin in hidden-layer connectivity supports the learning of complex action relationships while building upon the structured spatial representation from the forward pass.
19

---

These findings validate our standard implementation approach described in Section 3.2 where forward pass connections are fixed and only hidden-to-hidde weights remain trainable. The natural preservation of ring structure in trainable forward weights suggests that this topology is inherently beneficial for processing spatial information, while adaptable hidden weights enable the task-specific learning demonstrated in our experimental results, Section 4.2
Neuron Weights in RNN Hidden Space (Hidden Hidden)
Step: 0
Aedre Muito
fittares Pollls
Step: 300000
Step: 600000
Step: 1000000
bucuus
431
Figure 8: Development of hidden-to-hidden connections over time, demonstrating emergence of learned relationships between neurons beyond initial ring topology.
A.5 Attractor Dynamies Validation
To validate the preservation of CANN dynamics in our DL implementation, we conducted controlled
experiments examining the temporal evolution of neural states under dynamic input conditions. Ring attractors, as demonstrated in biological systems [18] 38), maintain stable activity patterns through recurrent connectivity that enables persistent neural states [16].
We present four experimental tables tracking hidden states and outputs across all eight neurons (DINNIDLL88) the ring attractor. Tables and 3 examine pre-training dynamics under a two- stimulus protocol with asymmetric (K₂ 0.1 K1) and matched (K₁ K2) input amplitudes, respectively. demonstrating the network's ability to maintain stable attractor states and transition smoothly between them. Tables 4 and 5) present post-training dynamics after 315,000 and 750,000 training steps in the OpenAl highway [2I]. revealing how task-specific learning modifies the ring attractor structure while preserving attractor properties.
20

---

Table 2: Temporal dynamics of DL basee ring attractor with asymmetric input cues (K2 0.1 K)). The network maintains stable bumps and executes smooth transitions despite aater-of-magiitaee differences in input amplitude between the first cue (centered at neuron 2. steps 1-6) and second cue (centered at neuron 5, steps 7-14). Bold values indicate peak activation at cach timestep.
Hid Out
1.38 0.67 44.98
1.38 0.72 46.02
1.38 0.60 43.54
1.38 0.54 42.11
1.38 0.53 41.92
42.12 12 1.09 0.42 33.26 0.93 0.37 28.55 1.09 0.42 33.26 1.38 0.54 42.12
0.15 0.43 12.74 0.15 0.54 15.17
2.11
1.15
3.34
4.61
4.78
4.60
4.48
DLN8 Out
1.52 0.74 49.85
1.52 0.80 51.00
1.52 0.67 48.23
1.52 0.60 46.64
1.52 0.59 46.43
1.52 0.60 46.65
1.46 0.60 45.40 40
1.57 0.61 48.04 1.52 0.60 46.67 1.46 0.59 45.02
1.52 0.61 46.86 1.57 0.63 48.37 1.59 0.63 48.72 1.57 0.62 48.30 1.52 0.60 46.77 1.46 0.58 44.89
1.57 0.63 48.37 1.52 0.60 46.79 1.46 0.58 44.84 84
1.57 0.63 48.39 1.52 0.60 46.4 1.46 0.58 44.83
1.57 0.63 48.39 1.52 0.60 46.80 1.46 0.58 44.83
1.52 0.60 46.80 1.46 0.58 44.83
1.46 0.58 83
DLN8
1.55 0. -0.05
0.15 -0.09
0.15 0.01
0. 0.06
0.15 0.07
0.15 0,06
0,06
Hid
In
0.15
In
Table 3: Temporal ring attractor dynamics with uniform input cues (K1 = K2). Bold values indicate peak activation at each timestep.
Out
0.58 36.86
77.38
0.42 33.27
41.92 1.09 0.41 33.06 0.93 0.36 28.35 1.09 0.41 33.06
1.13
3.48
4.79
4.97
4.80
4.68
In Hid Out
1.46 0.72 48.04
1.46 0.77 49.08
1,46 0.64 46.28
1,46 0.57 44.68
1.46 0.56 44.46
1.46 0.57 44.68
1.52 0.58 45.24
1.52 0.60 46. 46.80
0.49 34.
DI.77
DLN7
Hid
1.09 0.61 37.38 0.93 0.56 32.75 1.09 0,61
0.16 -0.10 1.37 0. -0.08
0.16 -..10 1.19 0.15 0.10
0.15 0.01
0. 0.07
0.07
0.07
0.06
In
1.09 0.58 36.86 0.93 0.55 32.47 1.09
43.54 1.09 0.49 34.74 0.93 0.44 30.05 1.09
42.11 1.09 0.42 33.27 0.93 0.37 28.56 1.09
0.15
0.15
0.
Out
15.42 0.16 0.43 12.89 0.16 0.37 11.68
3.61
4.94
5.13
4.96
4.84
In Hid Out
1.42 0.72 46.85
1.42 0.76 47.8
1.42 0.63 45.02
1.42 0.56 43.42
1.42 0.55 43.20
1.42 0.56 43.42
1.57 0.57 46.05
1.57 0.63 48 48.4
1.57 0.63 48 40
DL.6
DLN6
Hid
0.16 0.01
0.16 0.07
0.08
0.07
0.06
0.16
0.16
0.16
In
Out
1.75
1.34
3.70
5.01
5.19
5.02
4,90
DLN5 Out
1.52 0.74 49 49.85 1.46 0.72 48 04
1.46 0.77 49.08
1.46 0.64 46.28
1.46 0.57 44.68
1.52 0.59 46.43 1.46 0.56 44.46
1.57 0.62 48.25 25 1.52 0.60 46.65 1.46 0.57 44.68
1.52 0.63 47.31 1.57 0.60 47.09 1.59 0.58 46.74 74
1.57 0.62 48.28 1.59 0.60 47.82
1.46 0.58 44.86 1.52 0.61 46.81 1.57 0.63 48.38 1.59 0.64 48. ,8.98
1.52 0.60 46. 80 1.57 0.63 48.39 1.59 0.64 49.00
1.52 0.60 46.80 1.57 0.63 48.39 1.59 0.64 49.01
1.52 0.60 46.80 1.57 0.63 48.40 1.59 0.64 49.01
49.01
DLNS
Hid
2,43 0.16 -0.08
1.50 0.16 -0.10
3,70 0.16 0.01
4.97 0.16 0.07
0.08
0.07
0.06
In Hid
0.64
5.14 0.16
0.16
0.16
1.59
In
Out 1.38 0.67 44.98
1.38 0.72 46.02
4.96
4.84
In Hid Out
1.52 0.80 51.00
1.52 0.67 48.23
1.52 0.60 46.64
1.57 0.63 48
DI.N4
DLN4
Hid
1.38 0.60
1.63 0.65 50.12 1.73 0.69 53.34 1.63 0.65 50.12 1.38 0.54
1.38 0.53
1.63 0.65 50.14 1.73 0.69 53.37 1.63 0.65 50.14 1.38 0.54
0.65 17.51 0.15 0.70 18.60 0.15 0.65 17.75 0.16 0.54
3.02 0.16 +0.05
1.56 0.16 -0.09
0.16 0.01
4.82 0.16 0.07
0.16 0.08
0.16 0.07
0.16 0.06
In
DI.33 Hid Out
1.63 0.75 52.28 1.73 0.78 55.21 1.63 0.75 52.28
1.63 0.82 53.83
1.63 0.71 51.50 1.73 0.75 54.70 1.63 0.71 51.50
1.63 0.64 49.94 1.73 0.68 53.17 1.63 0.64 49.94
3.60
4.98
4.80
4.68
In Hid Out
1.57 0.76 51.29
1.57 0.82 52.55
1.57 0.69 49.82
1.57 0.62 48.23 23
1.57 0.61 48.03
1.52 0.61 46.99
46. 80
DLN3
3.13 0.15 -0,02
1.46 0.15 40.08
0.15 0.01
0.07
0.07
0.07
0.06
0.60
In
0.15
0.15
0.15
0.15
1.52
Hid Out
0.82 53.83 1.73 0.86 56.97
3.43
4.64
4.79
4,61
4,48
In Hid Out
1.57 0.76 51 51.29 1.59 0.77 51.84
1.59 0.82 53.15
1.57 0.69 19.82 1.59 0.70 50.43
1.57 0.62 48.23 1.59 0.63 48.85
1.57 0.61 48.03 1.59 0.62 48.64
1.529 1.59 0.63 48.86
1.42 0.63 44.93 1.46 0.64 46.11
1.42 0.59 43.99 1.46 0.60 45. 45.24
1.42 0.57 43.70 1.46 0.58 44.96
1.46 0.58 44.84
1.46 0.58 44.83
44.83
44.83
83
DLN2
DI.2
0.14 -0.02 2.72 0.15 -0,00
1.24 0.15 -0.08
3.28 0.15 0.01
0.15 0.07
0.15 0.07
0.15 0.06
0.06
1.46 0.58
0.58
1646
In
0.15
Hid Out
4.50
4.66
4.48
4.36
In Hid Out
1.57 0,82 52.55
1.57 0.62 48.25
1.42 0.57 43.61
1.42 0.57 43.58
1.42 0.57 43.57
1.42 0.57 43.57
1.42 0.57 43.57
DLNI
DI.N
0.14 -0.09
0.01
0.05
0.07
0.06
0.14 0.06
In
1.63
0.14
0.14
0.14
0.14
0.14
Ctr Arg.ax Arg
ArgMax
Step Cue CTRNN DI.
2
2
2
4
5
Step Cue CTRNN DL
2
2
2
2
5
Cir ArgM
2
2
2
2
2
2
2
3
3
4
5
5
5
2
2
3
5
2
2
2
2
2
2
2
5
5
5
5
3
4
5
6
7
9
10
11
12
13
14
2
3
5
6
9
11
12
13
21

---

Table 4: Post-training dynamics after 315,000 5,000 steps in OpenAl highway 21 Temporal dynamics of DL iL-based ring attractor with uniform input cues (K) = K2). Bold values indicate peak activation cach timestep.
Oul
1.52 0.36 41.32
1.46 -0.64 17.97 1.52 -0.26 27.89 89
1.46 -0.52 20.73 1.52 -0.20 29.20
1.52 -0.12 30.91
1.52 -0.12 30.87
1.52 -0.14 30.48
1.46 -0.14 29.10
1.46 -0.15 28.87
-0.14 29.08
-0.14 29.099.12 .700
-0.14 29.07
-0.14 29.06
-0.14 29, 79.06
1.59 -0.06 33.73 1.57 -0.60 21.42 1.52 -0.2 22.16 1.46 -0.14 29.07
In Hid Out
1.52 0.29 39.17
1.52 -0.19 30.46
1.52 -0.14 31.73
.4 1.52 -0.07 33.42
1.46 -0.43 23.61 1.52 -0.06 33.38
1.52 -0.08 32.97
1.46 -0.08 31.62
1.46 -0.09 31.38
1.46 -0.08 31.59
1.46 -0.08 31.58
1.46 -0.08 31.58
1.46 -0.08 31.57
1.46 -0.08 31.57
1.46 -0.08 31.58
DLN8
DLN8
Hid
Table 5: Post- trrinning dynamics after 750,000 steps in OpenAI I highway 21 Uniform input cues (K₁ = K2). Bold values indicate peak activation.
1.46
1.52 -0.52 22.15 15 1.46
1.46
1.46
1.46
25.15
Out
-0.33 24.79
1.46 -0.49 21.42
1.46 -0.50 21.11
1.46 -0.51 20.91
1.59 -0.08 33.16 1.57 -0.59 21.56 1.52 -0.51 22.40
1.46 0.05 33.28 1.52 -0.25 27.92 1.57 -0.05 33.67 1.59 -0.07 33.32 1.57 -0.59 21.53 1.52 -0.53 21.92
1.46 0.04 33.09 1.52 -0.25 27.98 1.57 -0.06 33.31 1.59 -0.06 33.61 1.57 -0,60 21.43 1.52 -0.52 22.19
-0.66 33.73 1.57 -0.60 21.43 1.52 -0.5 22.16
-0.06 33.73 1.57 -0.60 21.42 1.52 -0.5 22.16
Hid Out
-0.28 26.35
1.46 -0.56 20.52
23.18
23.94
1.46 -0.44 23.41
1.42 -0.29 25.76 1.46 0.16 36.41 1.52 -0.19 30.59 1.57 0.02 36.52 1.59 -0.03 35.6 58.68 1.57 -0.52 24.08 1.52 -0.44 24.92
-0.26 26.34 1.46 0.12 35. 1.52 -0.19 30.43 1.57 40.01 36.19 1.59 -0.02 35 8.84 1.57 -0.22 24.05 1.52 -0.46 24.43
1.52 -0.45 24.71
1.46 0.11 35.43 1.52 -0.19 30.56 1.57 -0.02 35.8 35.82 1.59 -0.01 36.26 1.57 -0.53 23.97 1.52 -0.45 24.67
36.27 1.57 -0.53 23.94 1.52 00.45 24.68
-0.45 24.67
1.52 -0.45 24.68
1.52 -0.45 24.68
DLN7
Hid
1.52 -0.52
DLN7
1.46 -0.46
1.46 -0.03 32, 98.98 1.42 -0.49 21.13 1.46 -0.42
1.46
1.42 -0.32 24.83 1.46
1.52
In
Hid Out
1.42 -0.39 22.46
1.42 -0.66 16.62
1.42 -0.64 17.07
30.4 40.46 1.42 -0.57 18.58
1.42 -0.58 18.29
30.26 1.42 -0.60 18. 18.04
1.52 -0.25 28.04 1.57 -0.06 33.31 1.59 -0,06 33.72 1.57 -0.60 21.46
33.73 1.57 -0.60 21.42
In Hid Out
1.46 -0.05 32.37 1.42 -0.58 19.14
1.42 -0.57 19.61
1.57 -0.17 31.98 1.52 0.04 35.73 1.46 -0.02 33.06 1.42 -0.51 20.82
1.57 -0.29 29.21 1.59 0.16 39.34 1.57 -0.19 31.68 1.52 0.02 35.45 1.46 -0.03 32.78 1.42 -0.53 20.56
1.46 0.11 35.43 .3 1.52 -0.19 30.49 1.57 -0.02 35.82 1.59 -0.01 36.14 14 1.57 -0.53 23.94
36.27 1.57 -0.53 23.93
1.57 -0.53 23.93
-0.53 23.93
DLN6
DLN6
1.57
In
Out 36.48
1.46 -0.10 29. 82
1.46 -0.15 28.88 83.89
1.46 -0.07 30.53
Hid Out
0.27 38.91
1.46 -0.09 31.42
-0.01 36.27
36.27
DLN5
DLNS
Hid
1.46 0.20
1.46 -0.08
1.46 -0.08
-0.06
35.83 1.59 -0.01
1.59 -0.01
-0.01
In
1.52 -0.2 28.04 1.57 -0.06 33.32 1.59
33.31 1.59
1.57 -0.06 33.31 1.59
In
1.46
1.59
1.59
Out
1.52 0.25 38.97
1.57 -0.28 28.48 48 1.52 -0.07 32.04
31.84
1.57 -0.24 29. .2 1.52 -0.02 33.05
1.59 0.10 37.18 1.57 -0.23 29.4 29.44 1.52 -0.02 33.15
32.89
1.46 0.09 34.03 1.52 -0.25 28.07 1.57 -0.03 33.98
28.04 04 1.57 -0.06 33.31
DLN4 Out
0.13 37.4 37.46 1.52 0.31 40.52
1.57 -0.21 31.02 1.52 -0.02 34.81
1.57 -0.23 30.54 1.52 -0.03 34.58
1.57 -0.18 31.96 1.52 0.03 35.62
1.46 0.11 35. 35.41 41 1.52 -0.19 30.56 1.57 -0.02 35.82 15.82212
35.88 35.82
35.82
DLN4
Hid
1.57 -0.30 27.99 1.52 -0.08
1.57 -0.25 29.16 1.52 -0.03
1.57 -0.06
In Hid
1.46 0.11 35.42 1.52 -0.19 30.56 1.57 -0.02
1.52 -0.19 30.56 1.57 -0.02
1.57 -0.02
In
Out
1.57 0.06 35.99
28.04
28. 04
Hid Out
-0.19 30.56
DLN3
DLN3
Hid
1.52 -0.25
1.46 0.04 33.07 1.52 -0.5
0.25
1.66 0.04 33.07 1.52
1.57
1.52
In
In
DLN2 In Hid Out
1.59 0.31 40.35
1.59 0.09 36.94
-0.39 25.95 1.59 0.04 35.93
1.59 0.09 37.02
1.59 0.09 36.97
1.46 0.04 33. 00
33.09
1.46 0.04 33.08
1.4 1.46 0.04 33.07
In Hid Out
-0.19 31.24 1.59 0.42 43.78
1.59 0.15 39.26
-0.32 28.51 1.59 0.11 38.19
1.59 0.16 39.38
1.57 -0.28 29.39 1.59 0.17 39.56
35.41
0.11 35.41
DLN2
-0.27 26.26 1.46 0.11
1.46
Hid Out
-0.26 28.82
-0.39 25.92
-0.35 26.99
-0.35 26.86
-0.66 26.69
1.42 -0.36 23.24
1.42 -0.33 23.82
-0.34 23.70
-0.34 23.75
-0.34 23.74
23.74
-0.34 23.74
-0.4 23.74
DLNI Hid Out
1.57 -0.31 28.47
-0.28 29.54
1.42 -0.27 26.21
-0.27 26.27
1.42 -0.27 26.26
-0.27 26.26
-0.27 26.26
DLNI
-0.34
1.57
1.57
1.57
1.57
1.57
1.57
1.42
1.42
1.42
1.42
1.42
1.42
1.57
1.57
1.57
1.42
1.42
1.42
1.42
1.42
In
In
DL ArgMax
ArgMax
8
2
2
2
2
2
2
4
5
5
5
5
5
5
DL
2
2
2
2
2
2
4
5
5
5
5
5
5
Cue
Cir
2
2
2
2
2
2
5
5
5
5
5
5
5
5
Step Cue
Ctr
2
2
2
2
2
2
5
5
5
5
5
5
5
5
Step
2
3
4
5
6
7
9
10
11
12
14
3
5
6
8
9
10
11
12
13
14
22

---

A.5.1 Experimental Protocol
We present four experimental tables that track hidden states and outputs across all eight neurons (DLNI-DLNB) in the ring attractor. The experimental setup consisted of an 8-neuron ring config- uration with parameters tuned for single stable bump formation: т 120.0 (temporal integration constant) and 9 = 22.0 (output scaling). The protocol involved an initial Gaussian input centered
on neuron 2 for t E [1. 6). followed by an instantaneous transition to Gaussian input centered on neuron 5 for 2 7. This experiment tests the network's ability to maintain stable attractor states and transition smoothly between them, as predicted by theoretical models of ring attractor dynamics [17]. Tables 2 and assess pre-training dynamics under this two-stimulus protocol with asymmetric (K2 = 0.1 K1) and matched (K₃- = K2) input amplitudes, respectively. Tables 4 and 5 Present post-training dynamics after 315,000 and 750,000 training steps in the OpenAl Highway environment (2T. revealing how task-specific learning modifies the ring attractor structure while preserving fundamental attractor properties.
A.5.2 Preservation of Attractor Dynamics
Table presents the temporal evolution of the DL-based ring attractor under the two-stimulus protocol with asymmetric input amplitudes where K₂ = 0.1 K1, following the input signal formulation in Eq. [ Despite the wrder-of-mmanittue difference in input strength, the network transitions between attractor states. During the initial phase (steps 1-6), neuron 2 maintains the highest output values (Out x 55.21), forming a stable activity bump. Following the input shift at step 7, the argmax tracks positions through intermediate neurons (2 + 2 - 3 4 + 5 over steps 7-11), exhibiting the characteristic temporal resistence of ring attractors [42]. By step 12, neuron stabilizes at Output = 5.19 despite the weaker input signal. Table presents the same protocol with matched input amplitudes (K₃ K2). The dynamics exhibit three characteristic phases consistent with CANN behavior [4. During the initial attractor formation phase (t E [1, 6)), the network maintains a stable activity bump centered neuron 2, with the argmax operation consistently identifying the correct peak location. The hidden state magnitudes remain comparable between the two stable phases (DLN2 Hidden R 0.77 at step I vs. DLN5 Hidden & 0.64 at step 12), confirming that the asymmetries observed in Table Table arise from input differences rather than structural biases. Following the input shift at f- = 7, the network demonstrates smooth state transition dynamics (t E [7. 11), where the activity bump migrates from neuron 2 neuron 5. The network subsequently converges to new stable attractor state (1 E [12, 14), settling into a configuration centered at neuron 5. This temporal lag reflects the 7 120.0 integration constant and should demonstrate the network's 's ability to maintain stable representations while integrating new information.
post-training dynamics after 315,000 and 750,000 training steps in the Highway environment, respectively. In Table 4 following the input shift step 7, the peak transitions from neuron 2 (step 2. Output R: 36.94) through neuron 4 (step 8, Output R= 33.67) to neuron 5 (steps 9-14, Output R 33..33.. The hidden states include negative values (e.g., DLN5 Hidden = -0.06 at convergence). indicating that task-specific learning has modified internal dynamics while maintaining the output behavior necessary for action selection. The attractor state of neuron is determined by the aggregate influence of all incoming connections through the hidden-to-hidden weights. As the hidden state undergoes transformation through y = f(WAA h1 + by). negative or non-maximal hidden values can produce the appropriate output behavior after passing through the tanh activation and scaling parameter 3. Table shows stabilization of these learned dynamics at 750k steps, with similar structural features confirming that the adaptations represent stable task-specific encoding rather than transient learning dynamics. Neuron 6. corresponding to the "move backwards" action in Highway, consistently shows lower activation values across both checkpoints (Output ~ 22.46 at step in Table 4. reflecting the penalty associated with this action.
A.5.3 Analysis
The preservation of attractor dynamics post-training aligns with findings in biological systems, where synaptic plasticity modifies connection strengths while maintaining functional circuit properties [37 24]. We may infer from these experimental results that our DL implementation preserves essential CANN dynamics while enabling task ssscispec adaptation through learning. The maintained bump migration and state persistence properties. combined with the evidence of spatial representations provided in the ablation studies in Section A.4 indicate that performance improvements stem from the ring attractor's inherent spatial encoding capabilities and attractor states rather than merely adding recurrent connections. The preservation of attractor-likk dynamics post-training, despite substantial weight evolution away from the initial Gaussian profile, reveals several computational mechanisms at work. The architecture maintains spatial awareness through the explicit ring topology: the circular arrangement of neurons preserves spatial relationships between actions throughout training, as evidenced by the natural preservation of distance-deppndent weights in the input-to-
23

---

hidden connections and the smooth bump migration patterns observed across all experimental conditions in Section[A.4 The ablation studies demonstrate that removing the ring structure causes performance to drop below baseline levels, confirming that spatial encoding provides functionally significant advantages over standard RNNs and feed-forwer forward networks. Simultancously. the learnable T parameter introduces temporal filtering dynamics that serve distinct functions across training phases. During early training, the results suggest that the architecture also functions as low-pass filter, where т creates momentum that reduces jitter in action exploration and enables the agent to reach distant state-space regions more efficiently.
A.6 Deep Learning Ring Attractor Recurrent Neural Network Modeling
As seen before, excitatory neurons are organized circular pattern, with connection weights between neurons determined by a distance bestance-weighted function mimicking the synaptic connection of biological neurons. Fig. illlstraaess the structured connectivity of the RNN, which mimies the circular topology of biological ring attractors.
FORWARD PASS
By
Ray
Dp
Dg
By
Ptn
Fet
Fax
Byr
LINEAR LAYER
TEMPORAL
INTEGRATION
HODDE
Vaus
Vasn
Vin
ng
Vana
Van7
Po
Van
nlg
Van.1
ng
n2
f1
SCALING
ACTION
ADTOR
ACTiD
HIDDEN STATE
HIDDEN-TO
b-Q
KEY
Use
Usa
Neuron
Learnable perameter
Use
Usa
Ring weighted connection
Connection
X
Operation
Action-value model outpul
Up77
Ug
Usa
Us,
Figure 9: RNN modeling ring attractor synaptic connections: The top panel shows the forward pass (input-to-hidden) as the agent output layer. The bottom panel depicts the hidden-to-hidden recurrent connection between inference time steps. Weighted connections illustrated for the sample neuron 72--
A.7 Deep Learning Ring Attractor Model Implementation Details
As presented in Scction 3.2 both the input-to-hidden connections (s) and hidden-to-hidee to-idde connec- tions (vv) are constructed using the distance-dependent weight functions defined in Eq. (12).

---

These equations establish the circular topology of the ring attractor and determine how information flows through the network. However, special case arises when dealing with neutral actions in certain Atari games, requiring a modification to the standard distance function.
A.7.1 Neutral Inhibitory Action Implementation
For games in the Atari benchmark with neutral actions (like 'no-op"), the ring attractor maintains its circular structure with a neutral action positioned centrally. This central position creates equal connections of strength to all other actions in the ring, as if it were a direct neighbor to each action simultaneously. The distance between the neutral action and any other action is fixed at 1:
11, dd(m n). d(m, n) = {
if m or n the neutral action otherwise,
(25)
where d(m, n) remains defined in Eq. 122 for all other action pairs. The weight matrices WI-HH and WH-H maintain the same exponential decay based on this distance function.
For example, in games like Seaquest or Asterix, this central positioning means the "no-op" action has consistent, strong connections to all directional actions. This arrangement preserves the spatial relationships between directional actions while ensuring the neutral action remains equally accessible from any game state. The constant distance of I to all other actions makes transitioning or from the neutral action as natural as moving between adjacent directional actions in the ring.
A.7.2 Deep Learning Double Ring Attractor Equations
For double ring configuration in our DL implementation as presented in the experiments, Section the weighted connections are defined as follows:
Let Vdouble E RR x2M be the complete input-to-hidden weight matrix for both rings, where N the number of output neurons per ring and M is the number of input features per ring. The matrix is structured as:
douubl
Vnn A
x xV
(26)
V21
V22
where V11 V22 V12 V21, K = 0.1 is the cross-coupling learnable parameter initialised to 0.1. This allows the network to learn the optimal strength of interaction between the two rings during training.
Developing from Eq. (12.. each ring maintains identical connectivity patterns, preserving the spatial relationships of their respective action dimensions, each submatrix Vij represents:
Walm,n (s) d(m,,)//
(27)
where d(m, n) is defined as per the forward pass weighted connections in Eq. (12).
Similarly, let Udouble E R22222 be the complete hidden-to-hiddee to-hidden weight matrix:
ddouble_
U 1I xU12
(28)
R3211
U22
For primary connections (Un1 and U22):
U h(v)Te -((mm)))/
(29)
where d(m, n) is defined as per the hidden state weighted connections in Eq. The complete forward pass for both rings given by:
12
L ( + ( = + - (1) ( T 1 U2z
Qs,, a) 3 tanh
( T 21 + (h)
(30)
where is the learnable time constant; 1 is the learnable scaling factor; $o(*t) is the feature representation of state sy; (v) is the previous hidden state; and x 0.1 is the coupling strength between rings.
25

---

The cross- coss-loup matrices (kV112, K VV21, UU2, and ^ xU21) maintain a circular topology similar the individual rings. A neuron at particular position in the first ring connects most strongly the neuron at the corresponding position in the second ring, with connection strength decreasing based on circular distance. This structured cross-coupling preserves spatial alignment between the two action dimensions while allowing semi-ind comi-independent operation through the learnable coupling factor k. The final output provides action-vas values for both action dimensions simultaneously, preserving spatial relationships within each ring while allowing weak coupling between them.
A.7.3 Extension to N Ring Configurations
The double ring implementation extends to R rings through a block matrix structure, where each ring encodes distinct action dimension. For I rings, the architecture uses block matrices Vintti R x and muut R RN x RN with diagonal blocks preserving individual ring dynamics and
off-diagonal blocks handling cross-ring interactions via coupling parameter K, as seen in Section A.7.2 While computational complexity scales ( selective coupling between only related dimensions creates a sparse structure with effective ( complexity. This makes the approach viable for complex action spaces where actions decompose into multiple indeinddeeenddnt planes, for example, in games that combine movement, combat, and resource- manareeemmnn dimensions.
A.8 Models and Environments Implementation
Table 6: Implementation details for ring attractor architectures across environments. The table shows the environment (Env); ring configuration (Ring); number of actions or continuous 1D action space (Actions); inhibitory neuron placed equidistant other neurons for "no action" term (Inhib); whether uncertainty estimation used (Uncert); the implemented model (Model); and type of Neural Network used (Type).
Game
Configuration Inhib.
Implementation Model
Environment
Ring
Actions
Uncert.
Тype
Highway
Single
Continuous
No
Yes
BDQNRA-UI -A
CTRNN
Mario Bros
Single
No
Yes
BDONRA-UA
CTRNN
Highway
Single
No
No
DDQNRA
DL-NNN
Mario Bros
Single
No
No
DDQNRA
DL-RNN
Alien
Double
18
Yes
No
EfTZeroRA
DL-RNN
Asterix
Single
Yes
No
EffZeroRA
DL-RNN
Bank Heist
Double
18
Yes
No
EffzeroRA
DL-RNN
BattleZone
Double
18
Yes
No
EiZeroRA
DL-RNN
Boxing
Double
18
Yes
No
EfiZeroRA
DL-RNN
Chopper C.
Double
18
Yes
No
EftzeroRA
DL-RNN
Crazy Climber
Single
Yes
No
ENZeroRA
DL-RNN
Freeway
Double
18
Yes
No
EfTzeroRA
DL-RNN
Frostbite
Double
18
Yes
No
EftZeroRA
DL-RNN
Gopher
Double
18
Yes
No
EnZerrRA
DL-RNN
Hero
Double
18
Yes
No
EnZeroRA
DL-RNN
Jamesbond
Double
18
Yes
No
EfiZeroRA
DL-RNN
Kangaroo
Double
18
Yes
No
EftZeroRA
DL-RNN
Krull
Double
18
Yes
No
EAZzroRA
DL-RNN
Kung Fu M.
Double
18
Yes
No
EfZZerRRA
DL-RNN
Ms Pacman
Single
Yes
No
EitzeroRA
DL-RNN
Private Eye
Double
18
Yes
No
E
DL-RNN
Road Runner
Double
18
Yes
No
EfTZeroRA
DL-NNN
Seaquest
Double
18
Yes
No
EfiZeroRA
DL-RNN
provide implementation details for both our models and the tested environments. For model implementations, EffzeroRA was applied across the Atari benchmark suite, while BDQNRA A-UA and DDQNRA were specifically implemented for Highway and Mario Bros. Table details the configuration of action spaces and ring architectures for each environment. The environments required different ring configurations based their control schemes, ranging from single-ring implementations for basic movement to double-ring setups for more complex action spaces that combine movement and specialized actions. Each ring topology was designed to preserve the natural
26

---

relationships between actions, with central inhibitory actions included where appropriate as "no action".
A.8.1
Computational Resources
For the Atari 100K benchmark (5] experiments, we utilized teae-peeformmnce computing cluster equipped with 10 NVIDIA A A00 GPUs (each with 80 GB memory), 512 GB of RAM, and 128 Intel Xeon CPU cores running 2.4 GHz. The cluster environment enabled parallel execution of experiments across multiple seeds and games simultaneously. with training times ranging from 8 to 10 hours per game, depending complexity.
For the other environments, Highway 21 and Super Mario Bros 15] we employed local worksta- tion with an Intel Xeon processor (28 cores, 2.1 GHz), 125 GB RAM, and dual NVIDIA RTX A5000 GPUs (total 48 GB combined memory). Training on the local machine typically required 6-12 hours per environment, with the CTRNN implementation incurring additional computational overhead as noted in Section 4.1 All experiments were conducted using Py PyTorch 1.12 with CUDA 11.6. The total computational cost is estimated at approximately 8,000 GPU-hours across all experiments and ablation studies.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and contributions in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes
Justification:
See Sections
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
28

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-
referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: See Sections
3
A.4.2
A.7
A.8 and link to code in Abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., s. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See link to code and implementation in Abstract.
Guidelines:
29

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips cttccc/ppuu. wwtppblcc//aaeeeccceeec....eccee caalcc/guies.ee (roodeSumissiiooooo..
for more details.
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Sections 4 A.8 and link to code in Abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: See Sections
A.3.2
A.4.1
A.4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support te main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
30

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section A.8.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors confirm that the research adheres the NeurIPS Code of Ethics and that anonymity has been preserved.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work presents biologically inspired reinforcement learning method evaluated solely in simulated environments, without application to real-world systems or human subjects. As such, it does not pose any identifiable societal risks impacts this stage.
Guidelines:
The answer NA means that there no societal impact of the work performed.
authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
31

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no risks for this matter.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Sections
A.4.2
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See Sections
A.3.2 and link code (anonymized assets) in Abstract.
Guidelines:
32

---

The answer NA means that the paper does not release new assets. Researchers should communicate the details the datasct/code//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
33


Reviews:
[{"review_id": "qE3oVif5Xg", "reviewer": "Reviewer_fh18", "summary": "The paper takes a standard model of ring attractors popular in neuroscience and applies it to action selection in reinforcement learning, with the idea being that ring attractors will capture the spatial structures well in the action space, and lead to better action selection and consequently improved rewards in reinforcement learning scenarios. With this insight the ring attractor is implemented by hand initializing a recurrent neural network policy that is fed as input the state features. Extensive empirical evaluation backs up this conceptual claim, with state of the art results and consistent improvements upon imbuing baselines with the ring attractor inductive bias.", "questions": "Questions\n- As far as I can understand, the DL approach starts with a ring attractor bias and then keeps the parameters of the input to hidden and hidden to hidden transitions learnable. If so, how much does the learning deviate from the original ring attractor dynamics to something else?\n- Could you give more insight as to why the performance improvements are so strong upon imposing this inductive bias on the action space? Could similar results be achieved by learning action embeddings instead that implicitly learn the implied spatial strcture in the action space?\n- Could you elaborate on where the inductive bias might not be right to use and instead hurts instead of helping performance?", "limitations": "No", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n+ The motivation for the work intuitively makes sense\n+ The work takes previous work on ring attractors and applies it to reinforcement learning, achieving strong empirical results\n+ The paper is well written and the results are nicely presented.\n\nWeaknesses\n- It would be useful to clarify why the addition of the ring attractor is supposed to help. The paper does a good job of explaining what a ring attractor is mathematically, and the dynamics associated with it, but why should it help for this particular RL scenario? Should it only help in continuous action spaces or could it also be useful for discrete action spaces? Is the idea that inputs are high dimensional and noisy, and one wants to be robust when picking actions for the control problem? More insight on this clearly spelled out would substantially improve the reach of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the careful review and particularly appreciate the reviewer's recognition of our research.\n\n## Addressing Weaknesses\n\n### Clarifying Why Ring Attractors Help: The Role of Spatial Inductive Bias\nRing attractors maintain stable spatial information representations, preserving action relations lost in traditional flattened action spaces. The fundamental mechanism operates through distance-dependent connections that encode spatial relationships between actions, with the attractor state evolving according to Eq. 2,3 for the CTRNN implementation and Eq. 12 for the DL approach.\nThe spatial inductive bias accelerates learning by explicitly encoding action relationships within the circular topology, enabling the agent to leverage similarities between neighboring actions in the spatial domain. This structured representation facilitates more efficient exploration and policy gradient estimation, as evidenced by our 53% performance improvement on the Atari 100k benchmark. Our analysis in Section A.4.3 demonstrates that when forward pass weights are made trainable, the network preserves the distance-dependent decay patterns throughout training, Fig. 7, suggesting that the spatial topology provides an inherently beneficial structure for action-value function approximation. This preservation suggests that the ring attractor has converged to an optimal spatial configuration that enhances learning efficiency.\n#### Evidence from Ablation Studies:\nSection A.4.1 presents the CTRNN ablation study where BDQNRA-RM randomly distributes the action space across the ring, disrupting the natural topology. Fig. 5 shows this manipulation eliminates performance gains, reducing results to baseline BDQN levels. From these ablation studies we may infer that the settling dynamics contribute to the observed improvements.\nThe DL ablation study (Section A.4.2, Fig. 6) removes the circular weight distribution, eliminating spatially-structured recurrence and effectively converting the ring attractor layer into a standard RNN. Performance drops in spatial environments like Ms Pacman and Chopper Command. Section A.4.2 demonstrates that removing the circular weight distribution in our DL implementation eliminates the spatial structure, resulting in performance degradation in spatially-oriented environments like Ms Pacman and Chopper Command (Fig. 6), further confirming the importance of the ring topology. Because the only difference from the full model is the absence of proper ring interactions, these results suggest that the spatial structure may drive performance improvements.\n#### Applicability Across Action Spaces:\nThe approach handles both continuous and discrete action spaces through unified spatial encoding. For continuous spaces, Fig. 2 demonstrates application to the Highway environment with 1D continuous circular action space. For discrete spaces, Table 1 shows improvements across the Atari benchmark.\n\n**Action:**  We will add at the beginning of Section 1 (line 32), following up from the mention in the abstract to the problem statement, this prior to our research:\n\n> \"While many action spaces possess inherent topological structure (Todorov et al., 2012), traditional neural networks represent actions as orthogonal vectors that ignore these relationships. Ring attractors bridge this gap by providing a neural substrate that preserves spatial ordering and similarity between actions. This is particularly relevant given recent advances showing that spatial structure in representations improves sample efficiency [2, 44] and generalization [3, 13], yet these benefits have not been systematically applied to action selection mechanisms themselves.\" \n\n## Addressing the Questions\n\n### Question 1: Learning Deviation from Original Ring Attractor Dynamics\n\n#### Standard Implementation Approach:\nOur implementation maintains fixed, input-to-hidden connections as described in Section 3.2.1, where weighted RNN connections include fixed, non-trainable, input-to-hidden connections (wI→H) to maintain the ring's spatial structure, and learnable, trainable, hidden-to-hidden connections (wH→H) to capture emerging action relationships as mentioned in lines 283, 284.\n#### Ablation Study Insights:\n\nSection A.4.3 analyzes model dynamics with both forward pass and hidden-to-hidden weights made trainable, which should address this question. The results suggest two patterns:\n\n**Forward pass preservation:** The forward pass connections preserve the ring structure over training time, with distance-dependent decay patterns maintained throughout the learning process, as demonstrated in Fig. 7.\n\n**Hidden layer adaptation:** The hidden-to-hidden connections develop specialized patterns that enable the encoding of environment-specific relationships between neurons in the hidden space while building upon the structured spatial representation from the forward pass (Fig. 8).\n\nThese findings may indicate that our standard implementation approach where forward pass connections are fixed and only hidden-to-hidden weights remain trainable is reasonable, as the network appears to preserve spatial topology even when given freedom to deviate.The learnable time constant τ and scaling factor β from Eq. 12 control the temporal dynamics and magnitude of the attractor's hidden state contribution over time. While we have not provided specific data points for the evolution of these parameter values, our appendix analysis demonstrates the contribution and stability of both hidden and feedforward connections in the DL implementation. Fig. 7 and Fig. 8 show that these connections do not collapse during training and maintain reasonable, consistent scales over time. Further insights on the parametrisation of the DRL agent can be found in the code link attached in the abstract as well. \n\n### Question 2: Understanding the Strong Performance Improvements\n\n#### Spatial Structure as a Contributing Factor:\nThe spatial encoding may translate into performance gains through action neighbouring. Our ablation studies suggest that randomized action arrangements degrade performance, which could imply that the spatial structure contributes to the observed improvements.\n#### Empirical Evidence:\nSection A.4.1,  A.4.2  shows that BDQNRA-RM with scrambled distance-dependent interactions and EffZeroRNN with removed circular topology result in performance in line with baselines, while maintaining proper spatial relationships yields the 53% improvement observed on the Atari 100k benchmark. The controlled nature of this experiment suggests spatial structure as a contributing factor.\n#### Complementary Uncertainty Benefits:\nAdditionally, the uncertainty-aware capabilities provide additional performance improvements beyond spatial encoding. Fig. 2 shows BDQNRA-UA with ring attractor and uncertainty awareness consistently outperforms both standard BDQN and BDQNRA without uncertainty.\n### Question 3: Limitations and When the Inductive Bias Might Not Help\n\n#### Evidence of Performance:\nOur experimental results suggest that ring attractors provide performance improvements across diverse RL tasks, with benefits extending beyond purely spatial tasks. Table 1 shows performance at or above baseline levels across all tested environments. For environments requiring multiple action dimensions, our double ring configuration (detailed in Appendix A.6.2) maintains spatial relationships within each action plane while allowing controlled interaction between dimensions through the coupling parameter κ, as demonstrated in games like Seaquest and BattleZone.\n#### Potential Limitations:\nThe approach would likely face limitations in action spaces with completely uncorrelated actions or very small action spaces where the ring structure overhead provides no advantage, e.g. action space with two discrete actions.\n\n**Action:** We will flag in Section 5 (line 382) additonal limitations:\n\n> \"The ring attractor approach may exhibit reduced applicability in discrete action spaces with cardinality |A| < 3 where the circular topology overhead exceeds potential spatial encoding benefits, and in action spaces where the action correlation matrix approaches the identity matrix, indicating absence of spatial dependencies or sequential relationships between discrete actions.\"\n\n#### Future Research Directions:\nWe agree with the reviewer that further work can be done to explore complex scenarios to put the inductive bias through further real-world tests. Which we are aiming to cover in a subsequent publication with deployment in continuous 3D real-world robotics tasks. This limitation has been outlined in our future work, Section 5."}, {"review_id": "bMgyZf7lMs", "reviewer": "Reviewer_n4bV", "summary": "This paper introduces a biologically inspired module for reinforcement learning (RL) policies based on ring attractor networks, motivated by their role in spatial cognition in the brain. The authors propose to incorporate a ring attractor mechanism as a plug-in feature encoder to enhance agents’ spatial awareness and decision-making. The module can be seamlessly integrated into existing actor-critic architectures. Experiments on the Atari 100k benchmark demonstrate notable improvements over baseline RL methods, particularly in tasks requiring directional memory and spatial inference.", "questions": "1. How does the method scale to continuous state spaces or tasks with higher-dimensional spatial structure ?\n\n2. How do the ring attractors perform under observation noise or partial observability?\n\n3. Does the ring attractor apply to the whole model?\n\n4. How does this compare to graph neural network or grid‑cell inspired architectures (existing spatial inductive bias)?", "limitations": "see above sections.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. It is interesting to introduce the use of ring attractors for spatial representation in RL. It is original and well-motivated by neuroscience literature.\n\n2. The proposed method yields consistent performance gains across multiple Atari games.\n\nWeaknesses:\n\n1. This paper is not easy to follow. I cannot fully understand the connection between the deep insights and the proposed method.\n\n2. It remains unclear what additional benefits the ring attractor component offers for uncertainty quantification, given that the final layer of the Q-function already employs a probabilistic formulation compatible with Bayesian linear regression.\n\n3. It is unclear what the key components are that result in the boost in empirical performance, action sampling or something else?", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough and constructive feedback and for the appreciation of our work. Below, we address each point raised by the reviewer:\n\n## Clarity and Connection Between Insights and Method\n\nLet us clarify the connection between the deep insights and the proposed method, starting from the beginning:\n\n**1. Biological Plausibility, Appendix A.1:** Ring attractors represent a computational principle where neurons are arranged in a circular topology to maintain spatial representations. Originally proposed by Zhang [45] for encoding heading direction and empirically discovered by Kim et al. [16] in Drosophila, these networks exhibit circular organization where head direction cells encode different spatial directions [36]. For our methodology, biological ring attractors integrate multiple sensory inputs through distance-weighted connections [46, 12] and maintain representational stability through excitatory-inhibitory dynamics [35], properties we implement through our distance-based weight functions in Eq. 4 and neural dynamics in Eqs. 2, 3.\n\n**2. Technical Translation of Neuroscience-based Principles, Section 3.1:** Our method translates these biological principles by mapping actions to circular positions Eq. 5 and implementing synaptic connectivity through exponential decay functions w(Em→En)=e^(-d²(m,n)), Eq. 4 that mirror biological connection strengths between spatially proximate neurons. The continuous-time dynamics in Eq. 2, 3 capture the biological property of attractor states while enabling transitions between related actions, implementing the spatial encoding observed in biological systems where similar spatial directions maintain stronger mutual influence through the circular network topology.\n\n**3. Implementation Approach and Motivation, Section 3.1, 3.2:** We developed two implementations to validate different aspects of biological translation: the CTRNN exogenous model in Section 3.1 provides biological fidelity through explicit excitatory/inhibitory dynamics in Section 3.1.1 and external Q-value processing in Section 3.1.2, enabling validation of ring attractor principles with uncertainty quantification in Section 3.1.3. The DL integrated model, Section 3.2, embeds ring topology within RNN weight matrices Eq. 12, Section 3.2.1 for deployment in DRL frameworks, maintaining the biological principle of distance-weighted spatial relationships and attractor states while enabling end-to-end training and scalability within existing DRL frameworks.\n\n**4. Performance Connection to Biological Insights, Section 4, Appendix A.4:** The spatial encoding translates biological advantages into performance gains through action similarity leverage, demonstrated by our 53% improvement on Atari 100k benchmark, Table 1, Section 4.3. Our ablation studies represented in Fig. 5, 6 show that randomized action arrangements degrade performance, confirming the bioinspired spatial structure is necessary rather than incidental. Following uncertainty principles [42], our Bayesian implementation, Section 3.1.3, incorporates confidence-based decision making, showing that the foundation provides a framework for encoding spatial relationships in action spaces.\n\n**Action:** We will include an introductory paragraph in Section 3 at line 105 as follows:\n\n> \"Ring attractors represent a computational principle where neurons are arranged in a circular topology to maintain spatial representations. Originally proposed by Zhang [45] for encoding heading direction and empirically discovered by Kim et al. [16] in Drosophila, these networks exhibit circular organization where head direction cells encode different spatial directions [36]. For our methodology, ring attractors integrate multiple cues through distance-weighted connections [46, 12] and maintain representational stability through excitatory-inhibitory dynamics [35]. This bioinspired structure has been repurposed in this line of research to explicitly encode the action space during the decision-making process, as illustrated in this section.\"\n\n## Uncertainty Quantification Benefits\n\nThe ring attractor provides distinct advantages beyond standard Bayesian formulations:\n\n**Spatial Uncertainty Propagation:** Unlike standard BLR which treats actions independently, our approach propagates uncertainty spatially across the ring Eq. 11, allowing uncertainty about one action to influence similar actions. BLR Eqs. 9, 11 yields per-action mean/variance; the ring uses σa as the width of each Gaussian in Eq. 5, so uncertainty directly modulates how broadly activity spreads across neighboring actions, not just the final argmax.\n\n**Action-Space Structure:** The ring topology enables uncertainty to be distributed according to action similarity, not just individual action confidence. This is particularly beneficial for spatial tasks where action relationships matter.\n\n**Empirical Evidence:** Fig. 2 shows BDQNRA-UA with uncertainty-aware ring attractors consistently outperforms both standard BDQN and BDQNRA without uncertainty, demonstrating the complementary benefits.\n\n\n## Key Performance Components\n\nWe believe our ablation studies in Section A.4 should identify the contributions of the key components to provide the boost in empirical performance:\n\n**Spatial Topology:**  Fig. 5 shows that randomizing action positions in the ring in the CTRNN-based model (BDQNRA-RM) eliminates most performance gains, indicating spatial structure is key in delivering faster and overall high learning performance.\n\n\n**Circular Connectivity:**Fig. 6 demonstrates that removing distance-dependent weights in the DRL agent, effectively converting the ring attractor layer into a standard RNN, reduces performance significantly in spatial environments like Ms Pacman and Chopper Command.\n\n## Specific Questions\n\n### Question1: Scaling to Continuous and Higher-Dimensional Spaces\n\n**Continuous Spaces:** Section 4.1 and Fig. 2 demonstrate successful application to OpenAI Highway environment with continuous 1D circular actions.\n\n**Higher Dimensions:** Section A.6.2 and Table 1 show our double-ring architecture for games requiring multiple action dimensions (e.g., movement + combat). The approach scales to R rings via block matrix structures, Section A.6.3, maintaining O(R) complexity through selective coupling.\n\nHowever, we agree with the reviewer that further work can be done to explore real-world complex scenarios. There is always room to grow further this piece of work, which we are aiming to cover in a subsequent publication with deployment in continuous 3D real-world robotics tasks. This limitation has been outlined in our future work, Section 5, where we explicitly identify \"continuous control tasks\" as a research direction.\n\n### Question 2: Performance Under Noise and Partial Observability\n\n**Future Work:** We agree with the reviewer that we have yet to evaluate systematic noise robustness, which we aim to fulfill through 3D real-world robot navigation tasks.\n\n**Action:** We will add in Section 5: \n> \"Additionally, the extension to 3D spatial navigation tasks presents an organic expansion of this work, these scenarios will naturally incorporate realistic sensor noise to the system.\"\n\nRegarding partial observability, we believe our benchmarks actually do test partial observability: Super Mario Bros uses limited scrolling windows preventing agents from seeing upcoming obstacles, while Atari games feature off-screen threats and limited sensor ranges.\n\n### Question 3: Ring Attractor Application Scope\n\nFor the CTRNN model, as these models are not trainable, it becomes an exogenous model that performs action selection as outlined in Section 3.1.2.\n\nThe ring attractor applies specifically to action selection with full integration through the DRL implementation. As described in Section 3.2.1, the ring attractor serves as the output layer of the DRL agent, processing action-value estimates while preserving their spatial relationships. The approach integrates fully within existing DRL agents and frameworks (DDQN, PPO, EfficientZero).\n\n### Question 4: Comparison to Related Spatial Architectures\n\n**Graph Neural Networks:**  While GNNs capture spatial relations through learned graph structures [3], they require learning spatial relationships during training, whereas ring attractors exploit an inductive bias that directly encodes the action space and drives the policy through explicit spatial embedding. Additionally, GNNs lack the temporal dynamics of ring attractors, missing the uncertainty encoding and attractor state stability that impact guide action selection.\n\n\n**Grid-Cell Inspired Architectures:** Grid-cell architectures [2, 10] encode environmental spatial locations and focus on spatial memory and path integration, rather than directly embedding the action space into the policy. Grid-cell representations operate at the environmental level, whereas ring attractors provide direct action-level encoding that guides the action selection process.\n\n**Action:** We acknowledge that grid cells is another biologically plausible mechanism applied in navigation, so because of several similarities we will add a paragraph in Section 2.2 covering the fundamental differences as follows:\n\n> \"In RL, grid cells and ring attractors serve distinct computational roles. Grid cells encode spatial state information (where you are) through hexagonal firing patterns, functioning as fixed basis functions for value functions and spatial generalization (Banino et al., 2018; Cueva & Wei, 2018). Ring attractors encode persistent states (directional and spatial bias) through recurrent dynamics that actively maintain information over time (Khona & Fiete, 2022). While grid cells provide emergent spatial input features, ring attractors are dynamical components that provide intrinsic spatial encoding for memory states (Kilpatrick & Ermentrout, 2013), that here we translate to encoding of the action space.\""}, {"review_id": "qjhmSKtj93", "reviewer": "Reviewer_umxu", "summary": "This paper introduces ring attractors—neuroscience-inspired circular neural structures—into reinforcement learning to enhance spatial awareness and uncertainty-aware decision-making. By encoding actions in a spatially continuous manner, the method improves learning efficiency and robustness. Implemented as both a recurrent neural network and a deep learning module, the approach achieves up to 53% performance gains on the Atari 100k benchmark and excels in spatially structured tasks like Super Mario Bros.", "questions": "1. How does your ring-attractor encoding compare, both conceptually and empirically, to existing spatial representations such as grid-cell encodings or attention-based spatial RL?\n2.", "limitations": "See the **Weakness** section above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- This paper proposes a novel method integrating biologically inspired ring-attractor circuits into RL algorithms. It embeds continuous circular action-space representations and decoding them via attractor dynamics with Bayesian uncertainty. \n- The method demonstrates significant empirical gains across multiple game benchmarks.\n- The paper is well-written and easy to follow.\n\n**Weakness**\n- The related work section largely lists prior methods without critically contrasting them to the proposed approach.\n- All experiments are in Atari or simple simulated navigation (Super Mario Bros, highway). The applicability to continuous, high-dimensional control is untested.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and suggestions, we are grateful you appreciated the work!\n\n## Related Work Critical Contrasting\n\nBased on the reviewer point, we would like to go a step further and provide deeper insights into what sets apart our current approach from existing literature.\n\n**Action:** We will extend the critical contrasting points throughout the literature review in Section 2 as follows:\n\n### Section 2.1 Spatial Awareness in RL\n\n**Line 62:** \"Although these approaches demonstrate the importance of spatial awareness in RL, they often lack the biological plausibility found in neural circuits.\"\n\n> **Updated Line 62:** \"Although these approaches demonstrate the importance of spatial awareness in RL, they rely on emergent learning to develop spatial understanding through comprehensive architectural additions, requiring extensive training to discover action relationships. In contrast, our ring attractor approach explicitly encodes spatial structure in the action space, providing spatial inductive bias that accelerates learning rather than relying on the RL agent to autonomously discover all critical action space relationships, which may result in suboptimal or incomplete spatial understanding.\"\n\n### Section 2.2 Biologically Inspired Machine Intelligence\n\n**Line 78:** \"Their approach demonstrated rapid learning and adaptation to new tasks, similar to the flexibility observed in biological learning systems.\"\n\n> **Updated Line 78:** \"Their approach demonstrated rapid learning and adaptation to new tasks, similar to the flexibility observed in biological learning systems. However, these approaches primarily focus on mimicking neural representations or learning dynamics, whereas ring attractors specifically encode spatial relationships in the action space itself, providing both biological plausibility and direct spatial awareness for action selection.\"\n\n### Section 2.3 Uncertainty Quantification\n\n**Line 98:** \"...incorporates efficient Thompson sampling and Bayesian linear regression at the output layer to factor uncertainty estimation in the action-value estimates.\"\n\n> **Updated Line 98:** \"...incorporates efficient Thompson sampling and Bayesian linear regression at the output layer to factor uncertainty estimation in the action-value estimates. While these methods treat uncertainty estimation as a separate module applied to existing architectures, our ring attractor approach integrates uncertainty through the variance parameters of Gaussian input signals, making uncertainty awareness an intrinsic part of the spatial action representation.\"\n\n### Section 2 Summary (Lines 99-103)\n\n**Line 101:** \"However, there remains a gap in integrating these elements into a cohesive framework.\"\n\n> **Updated Line 101:** \"However, there remains a gap in integrating these elements into a cohesive framework that simultaneously provides biological plausibility, explicit spatial encoding, and natural uncertainty handling within a single neural architecture.\"\n\nThese additions would transform the related work from a simple literature review into a more critical analysis that clearly positions the ring attractor approach relative to existing methods.\n\n## Experimental Scope and Applicability\n\nWhile we acknowledge testing primarily on Atari and navigation environments, we believe our experimental scope demonstrates broader applicability than suggested:\n\n- **Diverse Action Spaces:** Our experiments span discrete (8-action Mario Bros), continuous 1D (Highway environment), and complex discrete spaces (18-action Atari games) with multiple ring attractors as detailed in Table 2.\n\n- **Two Implementation Paradigms:** We provide both CTRNN-based (Section 3.1) and deep learning implementations (Section 3.2), with the DL version specifically designed for \"integration and comparison with existing DRL frameworks\" enabling broader adoption.\n\n- **Scalable Architecture:** Section 3.2.1 and Appendix A.6 walks the reader through the extensibility to N-ring configurations for higher-dimensional control, with our double-ring implementation Eq. 26, 30; showing how the approach scales to multi-dimensional action spaces.\n\n- **Performance Consistency:** Table 1 shows consistent improvements across 19 different Atari environments with varying action complexities, relationships and different types of decision-making tasks, suggesting robust generalization.\n\nHowever, we agree with the reviewer on the continuous control limitation. There is always room to grow further this piece of work, which we are aiming to cover in a subsequent publication with deployment in continuous 3D real-world robotics tasks. This limitation has been outlined in our future work, Section 5, where we explicitly identify \"continuous control tasks\" as a research direction.\n\n## Specific Questions\n\n### Question 1: How does your ring-attractor encoding compare, both conceptually and empirically, to existing spatial representations such as grid-cell encodings or attention-based spatial RL?\n#### Conceptual Differences\n\n**Grid Cells vs. Ring Attractors:** Grid cells [2] encode spatial positions in the environment potentially key for navigation algorithms composed of visual place recognition, path planning and other localization modules. While our ring attractors encode spatial relationships between signals, or in this case, actions. As noted in Section 2.2, grid cells \"emerged naturally in agents trained on robotics tasks,\" whereas our approach explicitly structures the action space itself upfront.\n\n**Attention Mechanisms vs. Ring Attractors:** Attention-based approaches [44] reason about spatial relations between entities, that are learnt over time in vast datasets or simulation environments to tune the self-attention layers. Increasing computational overhead, size and complexity of the DRL agent. On the other hand ring attractors maintain stable representations of action relationships through attractor dynamics, Eq. 1, 2, 3, 4; enabling the agent to distill information with little to no additional architectural and training overhead.\n\n**Empirical comparison:** We haven't found any major research line that applied grid cells to well-known DRL benchmarks. Additionally, these two approaches are viewed as operating at fundamentally different levels of the decision-making hierarchy, with grid cells providing environmental state representation and ring attractors providing action space organization. Grid cells have been tied to navigation tasks so far, while ring attractors could be applied to spatial decision-making problems. The comparison would be like comparing a GPS system (grid cells mapping spatial location) with a 2D action palette (ring attractors organizing spatially-related choices into a circular selection interface), both spatial, but serving different purposes.\n\n#### Architecture Differences\n\n**Topology & dimensionality:**\n- **Ring attractor:** 1D continuous manifold with periodic boundary; ideal for relationships between signals/actions, Eq. 5, 6, 7, 8.\n- **Grid cells:** 2D hexagonal tiling of state space; emergent in RNNs for localization [2,10]. Our focus is on action selection and policy building, not spatial localization per se.\n- **Attention RL:** learns soft relational representations over entities [44]; no hard prior on spatial continuity, relies on data to infer adjacency.\n- **Uncertainty handling:** Our design injects σₐ from BLR directly into the bump width (σₐ in Eq. 5), giving an interpretable link between epistemic uncertainty and action diffusion; existing spatial encodings rarely bind uncertainty to the topology this tightly.\n\n**Action:** We acknowledge that grid cells are another biologically plausible mechanism applied to RL navigation tasks, we will add the paragraph below in Section 2.2 covering the fundamental differences:\n\n> \"In RL, grid cells and ring attractors serve distinct roles. Grid cells encode spatial state information through hexagonal firing patterns, functioning as fixed basis functions for value functions and spatial localisation (Banino et al., 2018; Cueva & Wei, 2018). Ring attractors encode persistent states (directional and spatial bias) through recurrent dynamics that actively maintain information over time (Khona & Fiete, 2022). While grid cells provide emergent spatial input features, ring attractors are dynamical components that provide intrinsic spatial encoding for memory states (Kilpatrick & Ermentrout, 2013), which we translate here to the encoding of the action space during the action selection process.\""}, {"review_id": "5ijugTJ1TH", "reviewer": "Reviewer_nzPF", "summary": "This paper proposed a method to extend existing discrete reinforcement learning models with a ring attractor and a CANN-like structure. The method improves the performance of those models in the Atari 100k benchmark.", "questions": "1. In line 172, why $K_i=Q(s, a)$? \n2. In Equation (5), why does the parameter $s$ exist on the right-hand but not on the left-hand?\n3. How do you sample in line 257?\n4.  The time constant $tau$ seems very quick. What is the final $tau$?\n5. How does $tau$ participate in Equation 12?", "limitations": "This paper does not discuss this part, but I don't think it has a potential negative societal impact.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe idea to extend existing discrete reinforcement learning models with ring attractors is innovative, and the result suggests a performance improvement.\n\nWeakness:\n\nThe method is not well-explained, and the math is hard to follow due to inconsistencies and notations. Although the part about the ring attractors is introduced, the exact way to extend existing models, such as those shown in the experiments, is not clearly stated. The functionality of the network that forms the ring attractor is not clear. According to the appendix, the deep learning ring attractor functions like an extra RNN layer in addition to existing models. The dynamics of the ring attractor, which is its most important feature, is not supported or demonstrated in the experiments or the appendix.\n\nMinor issues:\n1. The differential notation is noted as $d$ in the paper. It should not be italic, otherwise it is mixed with $d$ in line 159.\n2. Line  159 and line 160: $d^2_(m,n)$ refers to difference concept. So as in Equation 12.\n3. max and argmax should not be italic.\n4. Equations in Figure 1 are not clear. For example, what is $argmax(dv/dt)$? Is it explained in the text?\n5. $theta$ is used for different concepts, including parameters of different models, but is not distinguished. It is really confusing.\n\nI also suggest that the authors should prove that the performance of the baseline models cannot result in a better performance by adding a typical RNN or FNN layer, to discuss the possibility that the performance improvement is due to more parameters.", "was_revised": true, "final_justification": "During the rebuttal stage, my questions in the discussion focused on examining whether the deep learning implementation of the CANN in this paper preserves the core properties of the original CANN, and whether the model truly exhibits \"spatial awareness.\"  The authors' responses have helped me better understand the actual role of the \"ring attractor\" mechanism in this work and the underlying reasons for the observed performance improvements.\n\nI appreciate the technical contribution, but I also would like to emphasize that the explanation of the core mechanism in the paper appears to be inaccurate or misleading. The model integrates equations inspired by CANNs into an RNN layer and attributes its effectiveness to \"Spatial-Aware Decision-Making.\" However, based on the discussion and evidence provided, this does not seem to be the true reason for the performance gains.\n\nAccording to the clarifications, the layer primarily functions as a low-pass filter during the early stages of training, which reduces jitters in action exploration and allows the agent to explore more distant regions more quickly. In the later stages, the layer behaves more like a general RNN with temporal integration, which helps the network better fit the Q-values. These dynamics, while useful, are not reflective of spatial representation or attractor dynamics in the conventional sense of CANNs.\n\nI recommend that the authors revise the manuscript to more precisely describe the role of the RNN layer, avoiding potentially misleading explanations regarding spatial awareness. A more accurate characterization would strengthen the clarity and integrity of the paper and improve its value to the community.\n\nI believe this paper has the potential to be accepted, provided the authors revise the manuscript to more accurately and clearly explain the underlying mechanisms. If this was a journal review, I would recommend a major revision, with the intention of recommending acceptance depending on the quality of the revision. However, given the current version and the fact that the authors have not yet fully recognized or addressed this issue, I am maintaining my current score.", "rebuttal": "We would like to deeply thank the reviewer for their detailed feedback on our work, especially on the methodology front.\n\n## Regarding the Strengths of the Article\n\nBriefly, we would like to point out that we have also conducted experiments with a 1D continuous action space, Fig. 2. The intent is to showcase the ability of the ring attractor to handle both continuous action spaces and discrete ones, the latter ones being used in well-established benchmarks such as Atari.\n\n## Regarding the Extension of Existing Models in the Article\n\nWe believe Section 3.2.1 provides guidance for practitioners. Algorithm 1 offers step-by-step dynamics loop for the CTRNN implementation with Eq. 5, 6, 7, 8, and 11 being the key equations to map fundamentals, while Section 3.2.1 presents the mathematical formulations for the DL architecture. Additionally, Appendix A.6 goes through the integration of multi-ring architectures with the DRL approach when in need to cover higher dimensional spaces, and Appendix A.7 walks the reader through the particular implementation for each environment evaluated in the article. \n\nFor further insights, \"plug and play\" implementation and a follow-up on how this is being implemented into a fully functioning algorithm, we refer to our available codebase in the link in the abstract where we show multiple implementations for the CTRNN implementation and DL-based implementation of the ring attractor. \n\n**Action:** However, we acknowledge that this information might benefit from being more prominently featured in the article. We will work on adding distilled pseudo-code from the implementation in the appendix.\n\n## Concerning the Demonstration of Ring Attractor Dynamics\n\nWe believe our paper provides substantial evidence for the importance of attractor dynamics through a set of sections in the article. Section 3.1 gives the differential equations for the excitatory and inhibitory populations Eq. 6, 7 that generate the bump-of-activity characteristic of a ring attractor. Algorithm 1 shows that on every decision (action selection) step, we iterate these equations to reach attractor state, ensuring the network actually settles before an action is read out.\n\nMost importantly, we believe several ablation studies deliberately disrupt the dynamics while keeping all other code paths unchanged as data points to support our statements empircally:\n\n- **CTRNN**: BDQNRA-RM randomly permutes actions around the ring (scrambling distance-dependent interactions). This results in Fig. 6, showing a glorified forward pass that doesn't alter baseline model performance.\n- **DL-RNN**: DL-no-ring replaces the exponential decay distance weight matrix with a standard initialization matrix (eliminating spatially-structured recurrence), causing again performance in line with baseline Fig. 6. Because the only difference from the full model is the absence of proper ring interactions, we believe these results constitute strong empirical support that the settling dynamics are present and causally useful.\n\nAppendix A.4.3 tracks the learned weight matrices, showing that forward (input→ring) weights keep their spatial profile throughout training, while recurrent weights adapt task-specific asymmetries, Fig. 7, when leaving the DL-based ring attractor distance-based weights trainable. The persistence of this profile indicates that the network consistently reaches and learns to exploit the intended attractor regime.\n\n## On the Characterization and Ablation of an RNN Layer\n\nWhile our DL implementation does utilize RNN architecture, it fundamentally differs from standard recurrent layers through its structured single/multi-ring circular connectivity and distance-dependent weights (Eq. 12). It modulates the input and hidden state through two parameters: the learnable time constant τ and the distance decay parameter λ. The key innovation lies not in adding computational layers, as shown in the ablation studies in Appendix A.4, but in encoding spatial relationships through biologically-inspired ring topology. \n\nThis structured approach preserves action relationships and enables the spatial encoding capabilities that drive our performance improvements. As mentioned above, Fig. 6 shows how the RL agent performs when distance-dependent weights are removed and the weights are set back to trainable. Effectively turning the ring attractor layer back to a standard RNN, when doing so, the DRL agent shows similar or slightly lower performance compared to baseline.\n\n## Addressing Formatting and Notation Issues\n\n### 1. Differential Equation Notation\nIn an effort to improve readability, we will remove italics from the differential equations to set them apart from the distance-based variable.  \n\n**Action**: Use `\\mathrm{}` to remove italics from differential notation in equations across the article.\n\n### 2. Distance Function Clarification\nRegarding lines 159-160 and Eq. 12, we respectfully clarify that d(m,n) refers to the same concept throughout: The discrete distance based on the number of neurons between two interacting neurons. The different formulations (`|m-n|` vs. `min(|m-n|, N-|m-n|)`) account for linear vs. circular distance calculations within the ring topology, both representing neuron-to-neuron distance.\n\n### 3. Mathematical Operators\nWe agree with the reviewer and thank them for the feedback. Argmax as operator should not be in italics. \n\n**Action**: Convert Argmax operation to roman format.\n\n### 4. Differential Equations vs. Argmax\nThe differential equations (dv/dt) describe how neuron activations evolve until the ring attractor reaches a stable state, while argmax(V) operates on the final settled activation values V = [v1, v2, ..., vN] to select the action corresponding to the most active neuron, Eq. 8. The dynamics (dv/dt) drive the system to convergence, and argmax(V) reads out the final decision. \n\n**Action**: In order to simplify understanding throughout the article, we will assume that Fig. 1 is at the final attractor (i.e., settled state). This will enable to swap argmax(dv/dt) for argmax(V) and point to Eq. 8, so it's easier for the reader to follow.\n\n### 5. Parameter Notation Consistency\nθ consistently denotes the learnable parameters of the DRL agent across the different methodologies and the appendix. \n\n**Action**: We will reserve θ for the full parameter set (e.g., the neural network in Section 3.1.3) and use subscripts for specific subsets within the DRL agent.\n\n## Specific Questions\n\n### Question 1: In line 172, why Ki = Q(s,a)?\n\nIn the original ring attractor formulation (Eq. 1), Ki represents the magnitude or strength of a given input signal i. By setting Ki = Q(s,a), we establish that the magnitude of a given signal is given by a state-action pair for a given action. Effectively this means we map input signals (i) to actions in action space (a), integrating the ring attractor as policy from Eq. 1 in Eq. 5. We will add a remark to highlight the substitution of input signal (i) for actions (a) and we want to thank the reviewer for the point. \n\n**Action**: We will signal this substitution at the beginning of Section 3.1.2 with the following sentence: \n> \"To integrate the ring attractor with RL, we reformulate the input signals by mapping Ki to the Q-value Q(s,a), effectively substituting input signal index i with actions a∈A.\"\n\n### Question 2: In Eq. (5), why does parameter s exist on the right-hand but not on the left-hand?\n\nWe originally used xn(Q) for readability since the summation ∑(a=1 to A) Q(s,a) makes the dependence on specific Q-values explicit. \n\n**Action**: We will revise to xn(Q(s,a)) for notational consistency throughout Eq. 5.\n\n### Question 3: How do you sample in line 257?\n\nThe sampling in line 257 refers to Thompson Sampling within an established Bayesian Linear Regression approach from the referred work by Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar, line 227. \"Efficient exploration through bayesian deep q-networks.\" There is also a practical implementation of this method in the codebase linked in the abstract and we would be happy to explain in detail how this method works with our approach if there are any further questions.\n\n### Question 4: The time constant τ seems very quick. What is the final τ?\n\nIn our CTRNN implementation, we set τ = Δt where Δt represents the discrete time step size. For our experiments, we used τ = 1e-3 s, with a simulation time of 0.05 s. This gives a total of 50 iterations to reach attractor state. These values were chosen empirically to optimize the performance of the CTRNN model, and were preserved across experiments (environments, seeds and ablation studies). \n\n**Action**: Add simulation time integration parameters to pseudocode in Algorithm 1.\n\n### Question 5: How does τ participate in Eq. 12?\n\nIn Eq. (12), τ appears in the input signal computation V(s)m,n = (1/τ)Φθ(s)T wI→H{m,n}. Here, τ acts as a scaling factor that controls the contribution of the input features to the hidden state. A smaller τ makes the network more responsive to immediate inputs, while a larger τ creates more integration over time. This is a learnable parameter allowing the network to discover the optimal balance between responsiveness and temporal integration for each specific task. \n\n**Action**: We will expand the existing sentence in line 297 to: \n\n> \"Here, τ acts as a learnable scaling factor that controls the balance between immediate input responsiveness and temporal integration within the ring attractor dynamics, with smaller values increasing sensitivity to current inputs and enabling faster adaptation, while larger values promote stability and smoother transitions between attractor states by integrating information over longer time horizons.\"\n\nWe hope these clarifications address the technical concerns raised and demonstrate our commitment to improve the quality of our article."}]

Author rebuttal:

Meta-review: (a) Scientific Claims and Findings 

The paper claims that integrating a neuroscience-inspired ring attractor structure into the RL action-selection process improves learning speed and accuracy by encoding spatial relationships between actions. It supports this claim with two implementations: a formal continuous time RNN (CTRNN) model and a practical implementation based on a typical RNN module. The primary finding is a significant performance improvement over baseline methods on the Atari 100k benchmark. Ablation studies seem to confirm that this performance gain is attributable to the specific circular topology of the ring structure, although why that is the case is not clear.

(b) Strengths 

The paper's main strengths are its novel model and strong empirical results. The idea of using a ring attractor to impose an inductive bias directly on the action space is simple, novel and well-motivated from a computational neuroscience poitn of view. The paper is also comprehensive, exploring both a formal model and a practical deep learning implementation, and includes thoughtful additions like uncertainty quantification and extensions to multi-dimensional action spaces.

(c) Weaknesses 

The main weakness identified during the review was *the lack of clarity regarding the methodology and mechanism*. Initial descriptions were confusing (had to go over it 2-3 times), relying  maybe too heavily on a neuroscience analogy without fully explaining its function in a DRL training loop. Discussion with Reviewer nzPF suggested the DL module's behavior might be better described as temporal smoothing aiding exploration, a nuance missed initially  - this still needs further understanding and validation. There’s still a disconnect between the motivation and the inspiration model and what is likely happening in practice, which is a concern that would need to be addressed.

(d) Reasons for Decision 

The paper received *tentative acceptance* due to its significant empirical results, obtained by a fairly simple mechanism and originality of approach inspired by neuroscience. However, concerns remain regarding clarity and mechanistic explanations. While partially addressed in the rebuttal, further revisions are needed to fully resolve these rectifiable issues, potentially requiring additional text refinement, explanatory details, or supplementary analyses.

(e) Discussion and Rebuttal 

The rebuttal period significantly improved the paper, satisfying three of four reviewers and increasing their confidence. Reviewers fh18, n4bV, and umxu found their concerns addressed regarding motivation, uncertainty quantification, and conceptual differences. The multi-turn discussion with Reviewer nzPF was crucial, revealing the DL module acts as a "low-pass filter," reducing action jitter. While nzPF maintained a borderline reject score due to perceived narrative contradiction, they recommended acceptance if this revised description was included - I tend to agree with this assessment.
Decision: Accept (poster)