RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers
Authors: Md Zesun Ahmed Mia
Malyaban Bal
Abhronil Sengupta
Venue: ICLR 2026 Poster (2026)
Keywords: Brain-inspired machine learning
Astromorphic transformers
Short-term Plasticity
Long-term PLasticity
Long-context sequence modeling
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Astrocyte-inspired LTP/STP guide memory compression and replay to make long-context Transformers practical

Abstract: The quadratic complexity of self-attention mechanism presents a significant impediment to applying Transformer models to long sequences. This work explores computational principles derived from astrocytes—glial cells critical for biological memory and synaptic modulation—as a complementary approach to conventional architectural modifications for efficient self-attention. We introduce the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), an architecture integrating abstracted astrocyte functionalities. RMAAT employs a recurrent, segment-based processing strategy where persistent memory tokens propagate contextual information. An adaptive compression mechanism, governed by a novel retention factor derived from simulated astrocyte long-term plasticity (LTP), modulates these tokens. Attention within segments utilizes an efficient, linear-complexity mechanism inspired by astrocyte short-term plasticity (STP). Training is performed using Astrocytic Memory Replay Backpropagation (AMRB), a novel algorithm designed for memory efficiency in recurrent networks. Evaluations on the Long Range Arena (LRA) benchmark demonstrate RMAAT's competitive accuracy and substantial improvements in computational and memory efficiency, indicating the potential of incorporating astrocyte-inspired dynamics into scalable sequence models.

Published as a conference paper at ICLR 2026
RMAAT: ASTROCYTE-INSPIRED MEMORY COMPRES- SION AND REPLAY FOR EFFICIENT LONG-CONTEXT TRANSFORMERS
Md Zesun Ahmed Mia, Malyaban Bal & Abhronil Sengupta
School of Electrical Engineering and Computer Science Pennsylvania State University University Park, PA 16802, USA
{zesun, (zesun.ahmed, mjb9906, sengupta aemppptt@pppsueeee.m
ABSTRACT
The quadratic complexity of self-attention mechanism presents a significant im- pediment to applying Transformer models long sequences. This work explores computational principles derived from astrocytes-glial cells critical for biologi- cal memory and synaptic modulation as a complementary approach conven- tional architectural modifications for efficient self-attention. We introduce the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), an archi- tecture integrating abstracted astrocyte functionalities. RMAAT employs recur- rent, segment-base processing strategy where persistent memory tokens propa- gate contextual information. An adaptive compression mechanism, governed by a novel retention factor derived from simulated astrocyte long-term plasticity (LTP), modulates these tokens. Attention within segments utilizes efficient, linear- complexity mechanism inspired by astrocyte short-term plasticity (STP). Training is performed using Astrocytic Memory Replay Backpropagation (AMRB), novel algorithm designed for memory efficiency in recurrent networks. Evaluations on the Long Range Arena (LRA) benchmark demonstrate RMAAT's competitive ac- curacy and substantial improvements in computational and memory efficiency, indicating the potential of incorporating sarrocyte-inspire dynamics into scalable sequence models.
INTRODUCTION
The Transformer architecture (Vaswani et al.] 2017) has become foundational for sequence mod- eling, particularly in natural language processing. A primary limitation, however, is the quadratic computational and memory complexity (O(N²) of its self-attention mechanism, hindering its ap- plication to very long sequences (Tay et al. 2020 Beltagy et al. 2020). The predominant research direction to overcome this focuses on moditying the Transformer architecture itself for greater ef- ficiency. Techniques explored include sparse attention patterns (Child al., 2019 Beltagy al.) 2020), linear attention approximations (Katharopoulos et al. 2020) Peng et al. 2021). state-space
models (Gu al. 2021 Gu & Dao 2023), and various recurrent structures (Peng al. 2023 Sun et al., [0233 Yang et al. 2023 Bulatov et al. 2022). Alongside efforts to improve architectural elficiency, research into brain-inspired computational principles gaining interest, driven by the potential for remarkable energy efficiency and novel processing mechanisms. However, similar to the challenges faced by conventional architectures, developing neuro-inspired learning approaches that robustly handle complex, long-range dependencies while being both computationally efficient and biologically grounded remains a significant hurdle (Bal Senguptal 2024). Addressing this challenge may require looking beyond purely neuronal models, many brain irsiiiinspired computing approaches focus predominantly on neuronal activity, often overlooking the computational roles of other critical cell types.
"Code Available: https: //gi http//gttttt/ttt wwt.///eroooooooo..abbooooo.. psu/ RMAAT git

---

Published as a conference paper at ICLR 2026
Among these overlooked elements are astrocytes, a type of glial cell increasingly recognized not just for support functions but for their active participation in modulating synaptic transmission, plasticity, and memory processes critical for learning (Gibbs et al. 2008; Bohmbacc et al. 2022 Perea al. 2009 Alberini et al., 2018). Given their established role in modulating temporal information and memory consolidation within biological circuits, we build on the premise that principles derived from astrocyte function are particularly well-suited to addressing the long-range temporal depen- dency challenges inherent in processing extended sequences. Despite their potential, astrocyte-based computational principles remain severely underexplored in deep learning.
This paper introduces the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), an architecture that integrates specific, computationally abstracted astrocyte-inspired mechanisms re- lated to temporal memory processing (inspired by astroeyte long-term effects) and attention modula- tion (inspired by astrocyte short-term effects) within a recurrent transformer framework. Our goal is to leverage these neuro-glial principles to create an efficient approach for long-context sequence pro- cessing. Within this emerging line of work, foundational studies have shown that tripartite synapses can implement Transformer self-attention, validated via weights extracted from pre-trained networks (Kozachkov et al. 2023), and have developed theoretical models of neuron astrrocye associative memory and capacity scaling (Kozachkov et al., 2025). Subsequent Astromorphic Transformer architectures (Mia al., 2025) have shown that these principles can be instantiated in standard- scale machine learning models. Building on this trajectory, our work scales these astrocyte-inspired mechanisms demanding long-context benchmark (LRA), where they yield competitive perfor- mance and improved memory efficiency in the long-sequence regime. The remainder of this paper organized as follows: Section X details our main contributions and positions RMAAT relative to prior work. Section 3 describes the RMAAT model architecture and its bio-inspired components. discusses limitations and concludes the paper.
Section presents experiments and results. Section 5
RELATED WORKS AND MAIN CONTRIBUTIONS
Significant research addresses the (O(N2) complexity and long-context limitations of standard Transformers (Vaswani et al. 2017). Early efficiency improvement efforts focused on sparse or linear attention approximations (e.g.. Longformer (Beltagy et al.) 2020), Reformer (Kitaev et al., 2020 Others incorporated recurrence via state caching or compression (e.g.. Transformer- XL (Dai al. 2019), Compressive Transformer (Rae al. 2019) and some utilized explicit mem- ory tokens to carry context between segments (e.g., RMT (Bulatov et al. 2022), Memformer (Wu al. 2020 More recently, highly efficient architectures like State- State-Space Models (c.g.. S4 (Gu
it al. 2021). Mamba (Gu & Dao ) based on continuous time systems, and RNN/Transformer
hybrids (e.g., RetNet (Sun et al., 2023). RWKV (Peng et al. 2023), GLA (Yang et al., 2023)
employing innovations like retention mechanisms or gating, have achieved strong results through sophisticated architectural and mathematical advancements. However, developing methods that integrate deeper biological principles, particularly for complex functions like long-term memory integration, alongside computational efficiency remains an ongoing challenge. Separately, within mologicalll-inspiree computing, astromorphic approaches have explored leveraging astrocyte prin- ciples (Kozachkov et al., 2023), particularly adapting attention mechanisms based on astrocyte nonlinearities and inherent plasticities (Mia et al. 2025). While valuable, these efforts have pri- marily concentrated on the attention component itself. The potential for utilizing computational principles derived from astrocyte temporal dynamics, such as those involved in long-term plasticity (LTP) related to memory formation and consolidation, to specifically address the challenge of long- range context propagation in sequence models remains largely unexplored. Complementary lines of work have investigated neuromodulated Hebbian plasticity in RNNs (Miconi et al.) 2020; Duan et al. 2023); in contrast, our model fixes the qualitative modulation structure and timescales from a neuron-astrocyte network simulation and only learns the downstream architectural parameters. To address the above-mentioned gaps, the main contributions of our work are:
(i) A Distilled Computational Macro Model:
We propose and utilize a novel macro model, dis-
tilled from detailed computational models of neuron-astrocyte LTP dynamics (Perea et al. 2009
Alberini et al. 2018), which serves the foundation for RMAAT's recurrent memory system. (ii) Memory Retention Factor for Segment-
Processing: Building on the macro model (i), derive novel Memory Retention Factor that bridges the biological abstraction to RMAAT's recurrent architecture, instantiating the macro model's saturation dynamics as a concrete compres- sion schedule for segment- based processing of memory tokens. This factor achieves biologically-
2

---

Published as a conference paper at ICLR 2026
motivated context compression. differing significantly from architectures reliant externalyy man- aged memory (Bulatov et al., 2022 Wu et al. 2020). (iii) Efficient AMRB Training Algorithm: We propose the Astrocytic Memory Replay Backpropagation (AMRB) algorithm, enabled by the model's memory structure, which significantly reduces the memory footprint and computational overhead compared to standard BPTT chunk-based backpropagation for recurrent training.
THE RMAAT MODEL
3.1
FOUNDATIONAL COMPUTATIONAL NEUROSCIENCE MODEL
RMAAT's core mechanisms are derived from computational models of the tripartite synapse (Bohm-)
bach al. 2022 Perea al. 2009 Alberini et al. 2018), describing neuron-astrocyte interactions.
We model key plasticity dynamics operating at different timescales, abstracting the principles into our framework.
To capture rapid synaptic adjustments and spatial context, we model synaptic facilitation (siz) between postsynaptic neuron and presynaptic neuron j. and the as- sociated short-term astrocyte process parameter (plz). Their dynamics are conceptually governed by interactions reflecting neuronal co-activation (0(z, ((ze))(())))))))) )), astrocyte modulation (vx(2, )), decay r"), and coupling between astrocyte processes. operating on faster timescale (Tx, T,). Simpli- fied representations highlighting key dependencies are:
Short-Teree Plasticity (STP):
dsij
x -Bsij + +0(z))((z,) + (p(z))
(1)
dt
dp T dt
- Σ Teme
Here, T, represent neuronal activity, (vf,,) represents local astrocyte modulation. In Equation 2 the summation term captures the influence of other astrocyte process activities (pkr- associated with neuron pairs k,, !) on the specific process ply- The coupling tensor Tiyki represents the concentration fluxes or strength of influence (e.g.. via calcium diffusion) between the astrocyte process associ- ated with synapse (i,j) and other processes associated with synapses (k, The magnitude of these fluxes typically depends on the relative spatial positions and distances between the interacting synapses within the astrocyte's domain. Thus, the dynamics of piy are modulated by the spatial context encoded in this flux pattern. The influence of these spatially dependent interactions on local astrocyte dynamics provides the biological mapping for how RMAAT computes relative positional information within its attention mechanism (detailed later).
Long-Term Plasticity (LTP): To model slower processes related to modulating temporal informa- tion and memory consolidation, we consider the long-term astrocyte process parameter (p(y). This variable integrates the effect of sustained synaptic activity (8ig) over a significantly longer timescale (TD T acting as form accumulating memory trace.
dp' Tp dt
x x---p', x((s))
The dynamics governed by Equation representing the integration of synaptic history (siz) over longer timescales via the ply variable, provides the conceptual foundation for our subsequent devel- opments. Specifically, we distill the principles captured by these LTP dynamics into computational Macro Model (Contribution 1). which we then operationalize for segment- based sequence process- ing by deriving a Memory Retention Factor (Contribution 2) that instantiates the macro model's saturation curve as concrete compression schedule for RMAAT's recurrent memory tokens. The detailed derivation, simulation results showing the characteristic behavior, and implementation of this memory system are presented in Section 3.3
The subsequent sections detail how RMAAT translates these principles into computational ar- chitecture, moving from biophysical simulation to a macro-model of STP/LTP and then to con- crete mechanisms for within-segment attention modulation (STP-like) and rross-segment recurrent memory integration (LTP-like) (Debanne Inglebert 2023 Bicknell & Latham 2024) Stasenko & Kazantsev 2023).
3

---

Published as a conference paper at ICLR 2026
3.2
CORE ARCHITECTURE AND PROCESSING
RMAAT processes sequences using a recurrent Transformer architecture built upon segmented pro-
cessing and a bio-inspired attention mechanism with spatial encoding of relative position.
3.2.1 SEGMENTED PROCESSING AND BIOLOGICALLY INSPIRED MEMORY TOKENS
To address the quadratic complexity bot- tleneck of standard selffattention over long sequences, RMAAT adopts a segmented processing approach. The input sequence divided into non-overlapping, contigu- ous segments of a manageable maximum length N Neeg The core RMAAT layers pro- cess these segments sequentially, rather than operating on the entire sequence at once. key element enabling long-range depen- dency modeling across these segments is the incorporation of dedicated Memory To- kens. Inspired by the capacity of biologi- cal systems, particularly astrocyte networks, to maintain and integrate information over extended periods (as abstracted in Sec Sec 3.1), these memory tokens serve as persistent, evolving state. Let the set of M memory to- kens the start of processing segment be denoted by mem,, These tokens are processed alongside the actual input tokens within the seg- ment using the mechanisms described below (See (3222 and See (.2.3). The output representations corresponding to these memory tokens after processing segment form the updated memory state, mem,41. which then passed as the input memory to segment & + 1. This recurrent flow, concep- tually illustrated in Figure [] allows contextual information within the memory tokens to propagate across segments. This mechanism differs from approaches like RMT (Bulatov et al., 2022) or Mem- former (Wu et al. (0220). which often rely externally managed memory mechanisms specific architectural additions for memory updates. In RMAÁT, the updates to these memory tokens are intrinsically linked to the bio-inspired dynamics derived from our computational macro model (de- tailed in Sec 333] involving a dynamically derived retention factor), aiming for a more integrated and computationally distinct approach to memory management. The processing within each seg- ment, which updates both sequence and memory token representations, relies on the astromorphic attention mechanism described below.
Recument Meroery Augmented Autromorphic Transformer (RMAAT
Recumentt Procesing
[+1
2+2 segrintt= 2
1 sagmant 3
t Tep regment
segment 1
RIMAT
RIMATT
RMAAT
RWAAT
STP
STP
STP
STP
Figure I: Conceptual illustration of RMAAT pro- cessing through time unrolling. Processing within each segment incorporates mechanisms inspired by STP. The recurrent propagation of astrocytic mem- ory tokens (mem) integrates context across many segments, drawing inspiration from LTP principles for persistent memory.
3.2.2 ASTROMORPIIC ATTENTION MECHANISM
Within each segment processed by RMAAT (as described in Sec the standard computationally expensive ( self eeff-attention is replaced by an efficient Astromorphic Attention mechanism. Its design draws inspiration from computational models of the tripartite synapse (Mia al.) [025] Kozachkov et al. 2023) and specifically abstracts principles from the STP dynamics outlined in Section 3.1 To implement this mechanism computationally, we conceptualize it using two-layer neuron- neuron-astrocyte network structure (input/hidden layer and output layer), as depicted abstractly in Figure 2 (Right). The mechanism operates in two consecutive modes within this structure: Write and Read. (See Appendix B for full details).
Let d be the model's embedding dimension (input/output layer size) and m be the hidden layer dimension. For a given segment t, the input X consists of the N sequence tokens (xx) concate- nated with the M memory tokens (mem)), resulting in a total of N = =Naeq + M tokens processed within the segment. First, the combined input tokens X E Nxd are linearly projected into Keys (K), Queries (Q), and Values (V) using learnale weight matrices WK, Wq dRdx (project- ing to the hidden dimension) and Wy dd (projecting
the output dimension), such that K = XWx x (Keys, R Nx"), Q = XW2 (Queries, R xxm), and V = XWy (Values, R Nx4) A non-linear activation function & (e.g., o(x) elu(x) 1, following (Katharopoulos et al. 2020
4

---

Published as a conference paper at ICLR 2026
Mia et al. 2025)) applied clement-wise to K and Q, yields activated representations Φ(K) and (Q), analogous to activations in the hidden layer (presynaptic neurons).
The Write Mode then computes effective synaptic weights and states within this network struc- ture, encoding context based on Hebbian principles and astrocyte modulation. These represent learned parameters or aggregated states within the network. Specifically: The Neuronal Heb- bian Weight component (H neuron E mRdd) captures the direct correlation between activated keys Φ(K) (hidden layer activations) and values V (output layer), representing baseline Hebbian plastic- ity summed across the N tokens (conceptually linked to the (xx))(xr;) co-activation term in Eq. This models the connection strength between the hidden (presynaptic) and output (post- synaptic) layers based on direct neuron-neuron
Prosubliiks
Write Mode:
W
interaction. The Astrocyte-Modulltee Hebbian Weight component (Hastro E Rmxdd incorporates the astrocyte's modulatory in- fluence, specifically integrating relative positional information (conceptually linked to the as- trocyte modulation term (p())) in Eq. J where Piz dynam- ics are spatially modulated). Building on prior astromorphic transformer work (Mia et al. 2025), it uses the activation of relative positional encoding matrix R (astrocytic parame- ter), @RR, derived from STP dynamics (detailed in Sec 3.2.3) to represent the influence of rel- ative positioning. This models how astrocytes modulate the hidden-to-outpu output layer connec- tion based on spatial context. Concurrently, the Presynaptic State (g E 1x abstracts the non-linear astrocyte re- sponse (e.g., calcium dynamics, denoted by C ~ Ca²+ in Figure 2 to the cumulative presynaptic (key) activity o(K) within the segment. It aggre- gates the activated keys o(k,) (where @(k)) is the t-th row of o(K)) over the segment length N and applies a non-lincarity controlled by parameter a. This state g encodes recent activation history in the hidden layer, influenced by astrocyte dynamics (a), relevant for feedback modulation in the Read Mode. These three com- ponents, representing learned weights (Hneuron: Hastro) and an aggregated state (g) within the network, are calculated as:
Encoder Layer
W
X
Fararr Netvoik
Input Layer
Hidden layer
--aah Output layer
Read Mode;
Aathovade
Sarvisedion
x
W
HR
Eedsctiing
Figure 2: Overview of the Astromorphic Attention mecha- nism, detailing the flow from input to output. This architec- ture replaces standard self-attention with a bio-inspired, linear- complexity mechanism that emulates the function of tripartite synapse. The process unfolds in three main steps.
(1) Input Projection: The input sequence X, which includes both the seg- ment's tokens and the recurrent memory tokens, is linearly pro- jected into Key (K), Query (Q), and Value (V) matrices using learnable weight matrices WK. WQ. and Wv. (2) Write Mode (Context Encoding, brown path): This phase aggregates and encodes contextual information across the entire input segment, modeling how astrocytes integrate signals over a local neighbor- hood. It computes two Hebbian weight matrices: the Neuronal Hebbian Weight nHeurrom). ), capturing direct key-value correla- tions, and the Astrocyte-Moduulatee Hebbian Weight (Hastro), which is modulated by the astrocytic parameter R--a Iearnable matrix that abstracts the principles of lestance-depedeed spatial coupling quantified by the tensor Tizkı in the underlying neuro- science model (Sec. (32.3) andd introduces spatial context. To- gether, these Hebbian weights define effective synaptic connec- tions from the hidden layer (h) to the output layer (L) depicted in the figure. Concurrently, the mechanism computes a presynap- tic state g. which represents the astrocyte's integrated response overall neuronal activity and computationally abstracts the dy- namics of intracellular calcium concentration (Ca2+). (3) Read Mode (Context Retrieval, red path): This phase uses the coded context to generate the final output for each token, mod- eling ietrocyte-mediated feedback. First, each token's unique Query vector is used to compute dynamic Presynaptic Plas- ticity Feedback Factor (P), which acts query-speeific feed- back signal. This factor then modulates the combined Hebbian weights (H- Hnewron + Hastro), and the resulting modulated weight is applied to the queries to produce the final output L at the output layer. The full equations are detailed in Section 3.2.2
5

---

Published as a conference paper at ICLR 2026
N g Σ (1e))
H newrron 1 772
H Hastro 1 m
_
(4)
The Read Mode uses the current queries Q (projected from input X) ) retrieve the context encoded during the Write Mode in the combined Hebbian weight H = Heuron + Hastro- This retrieval is modulated by an satrocyte-inspire feedback mechanism operating within the network structure. First, an interaction strength C' E I between the currently active queries 6(Q) (hidden layer activations from queries) and the cumulative presynaptic state g (astrocyte state abstraction) cal- culated as C ((Q)g" This represents the calcium response evoked by the presynaptic action po- tential (Q). Inspired by biological modulation (e.g., saturation) (Mia et al. 2025), a feedback factor P is derived, typically modeled as inversely related to this interaction strength, i.c., P 1/C. This represents the presynaptic plasticity decoded by the query (Q). The combined Hebbian weight ma- trix H is then modulated element-wise (Hadamard product @) by this feedback factor P. The final weight H @ P defines the synaptic weight between the hidden (h) and output layer (L). Activated queries ((Q) retrieve the relevant context by multiplying this modulated weight matrix (HI @ P). Finally, a standard residual connection adds the original input X to compute the Final Attention Output (L), which represents the final activation of the output layer for this attention block (further details are provided in Appendix B on how L maps to self- att-ntoent in transformer):
=QQ)(( O P) +
(5)
The resulting L E RNxd represents the updated token representations for the segment. This com- putation achieves O(N) complexity because the intermediate context aggregates (H and g) are computed once per segment with dimensions independent of N, and the final steps involving the N- dimensional query matrix (QQ) consist of operations like matrix-vector products that scale linearly with N, avoiding the quadratic cost of standard attention. The output L typically proceeds through standard subsequent layers like Feed-Forward Networks (FFN) and Layer Normalization within the overall Transformer block structure (Figure 2 left).
3.2.3 BIOLOGICAL GROUNDING OF RELATIVE POSITIONAL ENCODING BY STP DYNAMICS
Effective attention mechanisms in transformer architectures often benefit from incorporating rel- ative positional information to understand sequence order (Shaw et al. 2018 Mia et al., 2025) Common implementations define a base distance matrix for instance, using an exponential decay Pij exp(=I posyll x scale), where pos, and pos, represent token positions and scale is a tunable hyperparameter controlling the spatial range of influence. This base distance informa- tion is then transformed using learnable projections to compute a final positional encoding matrix R E Xm, with specific implementation details discussed in Appendix B and following prior works like (Mia et al. 2025). While such methods are computationally effective, they often lack direct biological correspondence. Our work seeks to provide this biological grounding by mapping the concept of relative positional encoding to principles observed in simulated astrocyte Short- Term Plasticity (STP) dynamics, particularly the role of the concentration flux tensor Tijki-
Our computational neuroscience simulations (Sec 3.1 Appendix C investigate spatial interactions among astrocyte processes. These simulations incorporate the distance-dependent coupling tensor Tiss (Eq. (2). reflecting how influence between processes diminishes with distance, akin to biolog- ical signaling like calcium diffusion (Wade et al. 2011 De Pittà et al. 2009) (the Teya-mediated coupling effectively defines "many-neuron synapses" that couple activity across multiple synapses within an astrocyte's 's domain Kozachkov et al. (2025)). In our simulations, this same coupling pro- duces spatially structured activity the astrocyte processes (p'z): processes near activity centers exhibit higher peak and more sustained activity than peripheral ones, reflecting their stronger inte- grated neighborly coupling.
This inherent encoding of spatial relationships within simulated STP dynamics provides strong biological rationale for incorporating a similar distance-based relative positional information scheme in our Astromorphic Attention. We translate this observed principle into the Astrocyte-Modulatee Hebbian Weight (Hastro) component via the term (RR), using the positional encoding matrix R (computed as described in the first paragraph). Calculating Hastro .,
1 L.)()) (Eq. 4 thus integrates a form of spatial context whose use is directly motivated by its analogy to simulated astrocyte STP behavior. This offers flexible, learnable, and catoogicalll-roondee method for

---

Published as a conference paper at ICLR 2026
incorporating relative positional context, distinct from standard approaches lacking this neuro-glial justification. Having addressed this spatially-infore component of attention, we now turn to the temporal memory mechanisms essential for processing long sequences.
3.3 ASTROCYTE-INSPIREE MEMORY MECHANISM
To effectively model long-range dependencies across the segments processed by RMAAT (Sec I.2.I. we require mechanism that not only propagates context but does so efficiently, re- flecting biological principles of memory consolidation. We draw inspiration from the Long- Term Plasticity (LTP) dynamies associated with astrocytes, particularly the behavior of the long-term astrocyte process parameter (p', in Eq. 3). which integrates synaptic activity (8ij) over extended timescales (T.).
Computational Macro Model (Contribution 1): We leverage the detailed computational neuroscience model of neuron-astrocylt interactions (Sec[.]) to un- derstand the principles underlying LTP. Simulations of
Symaplic Fart seciiaation and Astrocyto LTP Dynamics pl.
LTP
Adbor unt
this detailed model reveal essential characteristics of the LeP-related state (p'y): gradual integration of in- formation over successive Short-Term Plasticity (STP) cycles, continuous accumulation across these cycles, and eventual saturation. Figure 3 illustrates this sim- ulated behavior for 3 3 neuron network over 300 seconds (encompassing six 50s STP cycles). We inte-
30
100
1.5
200
250
300
Time 34]
Figure 3: Simulation of the computa- tional neuroscience model (3 x 3-neuron network (9 connections), 300s total time, x 50s STP cycles: STP cycles are re- every 50s in the 300s simulation) illus- trating temporal integration for astrocyte- inspired memory. Dashed lines show the long- Nong-term astrocyte process (p(y)) integrat- ing information and gradually saturating across STP cycles. Solid lines show the faster synaptic facilitation dynamics (8iz) within each STP cycle.
distill these observed Sharacteristics-tempee gration and naturation-ints computational macro model that captures the emergent dynamics of LTP- based memory consolidation at abstract level, inde- pendent of specific architectural choices.
From Maero Model to ML Archttecture (Contribu- tion 2): While the macro model captures the biological principle of saturating memory integration, does not directly prescribe how to implement this within a re- current transformer processing sequences segment-by- segment. Contribution 2 bridges this gap by deriving Memory Retention Factor that instantiates the macro model's saturation curve as concrete com- pression schedule for RMAAT's memory tokens in segment- processing.
To construct this factor, we normalize the macro model's total memory capacity (the integrated LTP sig- nal at saturation) to unit, then compute each segment's 's fractional contribution. Formally, for sequence with T segments, the Memory Retention Factor for segment I is:
numbrr STP cydee
number STP sydee
Memory Ratertonn
number STP cydes
03
Arl
RetentionFactor((.)
(6)
011
App
Sagments Cycles)
where Apt represents the incremental increase in the simulated LTP state during segment f, obtained by run- ning the LTP macro model (Eq. 33 for T segments and measuring Ap! =
Figure 4: Memory Retention Factor de- rived from simulating the LTP macro model for different total sequence lengths (represented as total number of STP cy- cles from 2 to 8). The factor decreases per segment as the total sequence length increases, implementing adaptive, bio- inspired context compression.
Teyote being pp:==p'(e p'(( (I-Tayde) p ((1-)) -eyyee) Teyele), with duration of one STP cycle. Figure ill- lustrates the resulting factors for sequence lengths of 2, 4, 6, and 8 segments, showing that each subsequent segment contributes diminishing fraction fraction-enablee enabling adaptive compression where the model adjusts reten- tion based on anticipated sequence length, mimicking biological resource constraints.
Application to Memory Tokens: This Memory Retention Factor is applied directly to RMAAT's persistent Memory Tokens mem, Ma (Sec 3.2.1). which carry context across segments. As

---

Published as a conference paper at ICLR 2026
tokens updated within a segment via Astromorphic Attention produce the next state mem++ the factor corresponding to the current segment number and total sequence length (Figure 4 scales the updated state (e.g., m RetentionFactor(t. T X mem see Algorithm I in Section 3.4 for the full pseudocode and Appendix D for additional implementation notes). This implements the adaptive compression dictated by the LTP macro model principle, ensuring memory remains bounded by gradually compressing older information. This contrasts with architectures like RMT (Bulatov et al. 2022) that often rely on fixed-size external memory slots updated via stan- dard mechanisms, lacking this specific bio- iio-inspieed, adaptive compression rationale derived from LTP dynamics. This integrated, astrocyte-inspiee memory system not only manages long-range context efficiently but also enables the resource-efficient AMRB training algorithm detailed next (Section 3.4.
3.4 AMRB TRAINING ALGORITHM
Training recurrent architectures on long sequences via standard BPTT is often memory-Prohiibitive due to storing activations for the entire sequence length. To overcome this while leveraging RMAAT's unique memory structure, we introduce the Astrocytic Memory Replay Backpropaga- tion (AMRB) algorithm (Contribution 3), an efficient training approach inspired by techniques for recurrent networks (Bellec et al.) 2019; Meng et al. 2023)The core idea of AMRB is to avoid storing all intermediate activations within each segment during the forward pass. Instead, it leverages the persistent, compressed Memory Tokens (mem; described in Section 3.3 During the forward pass through Teeg segments, only the sequence of memory token states (mem), meme, me.TT....] passed between segments is stored in replay buffer. During the backward pass, gradients are com- puted segment by segment. To calculate the gradients for segment t. the algorithm first retrieves the initial memory state mem, from the buffer. It then recomputes the forward pass for segment I only, starting from mem, and using the input tokens IG. This recomputation generates the neces- sary activations for calculating local gradients within segment t. The gradient flowing back from the subsequent segment t + 1 (which is initialized using the stored mem)++) is then backpropagated through the recomputed segment t, including the update path for memy, This process repeated backward for all segments. Algorithm| beloo provides detailed pseudocode description. The "re- play" aspect refers this recomputation of the forward pass for each segment during the backward pass, using the stored istrocyle-inspired memory state as the starting point.
Algorithm 1: Astrocytic Memory Replay Backpropagation (AMRB)
Input: rollout = (a, ..., ær.]
List of input tokens for T time steps.
Imput: my: Initial memory state (inpat to step i = 1).
Output: m: Updated memory (output after step T). Initialize replay bur + [] Append my to replay buf berf
// Store initial imput state
/x Posward Pass
x/
for t= to de
mii1 Model(as, ms):
// Compute intermediate state (no grad) // Apply retention factor
me+1 + RetentionFactor(t, T Seguenngm x m//11
itt < T then
L Append m.+1 to replay buffer: 1/ Store input state m441 next step's recomputation
/- Backward Pass Initialize Vmr++1 +0
11 Iail gradient Ioz stale after last step
for = To do
Retrieve m, from beplay.baffer
// Get input state for segment t 11 Recompata segment (track grada)
10
m/+1: ox <- (rdel me):
11
Crmpuute loss La <- loss function(oe):
// Compute loas for current step
12
Perform backpropggation: L, backwardO:
11 Compute pazam grads 8L/86,, etc.
13
mi , graawalllgaareenen = Vm 1. retain.graph-TTue):
// Compute Vm, via chain rule
14 Save m7 for the next rollout's update
AMRB offers significant memory efficiency and potential speed advantages. Unlike standard BPTT, which stores extensive activations, AMRB only caches the compact set of M memory tokens passed between the Teeg segments. Since M is typically small, the memory footprint drastically reduced. While AMRB involves recomputing activations during backpropagation, the associated memory saving often outweighs the recomputation cost for very long sequences, potentially leading to faster overall training (further details in Section 4 compared to standard BPTT.
adaptive memory compression system. This is realized through the Memory Retention Factor derived from simulated astrocyte I.TP, provid-
In summary, RMAAT introduces a novel
8

---

Published as a conference paper at ICLR 2026
ing principled, non-leamed method for compressing and propagating context between segments. Our core hypothesis is that this principled compression enables a more efficient training strategy. By structuring the flow of information into compressed set of memory tokens, we can forgo backpropagation through every token-the source of high memory costs in standard BPTT used in RMT and instead use our highly memory-efficient AMRB algorithm. Our ablation study in Section 4.2 validates this critical synergy: removing the compression causes a significant accuracy drop. This demonstrates that our bio-inspired compression is crucial for making the memory-saving AMRB algorithm effective, and this combination is directly responsible for RMAAT's gains in both efficiency and accuracy.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP AND RESULTS
Benchmark, Setup and Baselines: We evaluate RMAAT using the Long Range Arena (LRA) benchmark (Tay et al. 2020). Models are implemented in Py and trained from scratch (details in Appendix E.. We evaluate RMAAT against the standard Transformer and a selection of promi- nent efficient Transformer models. While recent works have ventured into alternative frameworks like State-Space Models (SSMs) Gu et al. 2022]); Gu & Dao (2023). we include these established efficient Transformers as they represent direct lineage of architectural modifications for efficiency, providing relevant context for RMAAT's approach which prioritizes deeper biological plausibility over purely mathematical or structural innovations. For a focused comparison of its recurrent and bio-inspired elements. we include key iso-architecture baselines. These are: Astromorphic Trans- former (AT) (Mia et al. (2025). a non-recurrent model with astrocyte features but missing RMAAT'S recurrence and memory; Recurrent Memory Transformer (RMT) (Bulatov et al., 2022). which pro- cesses segments recurrently with memory tokens but uses standard attention and lacks RMAAT's specific memory compression training; and Recurrent Linear Transformer (RLT), based on the Linear Transformer (LT) (Katharopoulos et al., 2020), implemented with recurrent structure and memory tokens RMAAT but without RMAAT'S 's specific memory retention factor, AMRB train- ing, or the enhanced positional encoding and non-linearity found in (Mia et al. (025). These latter models serve as important iso-architecture baselines isolate the effects of RMAAT's contributions.
Table 1: Accuracy and Memory Comparison on Long Range Arena (LRA) Benchmark Tasks.
Model
Listpps (2
Text (48)
Retrieval (8K)
Image (1K)
Patishoder (1 K)
Average
Mom."
Aoc(5)
Mem."
Aoc.S)
Mom.
Aoc(S)
Mem
Aoc.(S)
Mem:
Acc.
Transformer
Marwani er ml
2017
36.4(1)
4.7 64.3(1)
6.7 57.5(1)
5.2 42.4(1)
7.8
71.4(1)
5.4
54.4
Sparse
(1)1(1)
63.6(1)
69.6(1)
44.2(1)
71.7(1)
Longformer
Baaap
35.6(1)
62.9(1)
56.9(1)
42:2(1)
09.7(1)
Linforner
Wain mt
A147
35.7(1)
63.9(1)
52.3(1)
38.6(1)
76.3(1)
51.4
Reformer
Kitey 1:
2027
37.3(1)
66.1(1)
63.4(1)
38.1(1)
08.5(1)
50.7
Bighird
Zareer
36.1(1)
64.0(1)
59.3(1)
40.8(1)
74.9(1)
55.0
16.1(1)
4.7
65.9(1)
57
(3.1(1)
39
42.3(1)
6.2
76.3(1)
6.2
50.5
Percomer
[
Livemmask
2020
18.0(1)
65.4(1)
63.(11)
42.8(1)
7T.1(1)
51.4
Les cf mt 3721
35.3(1)
65.1(1)
58.6(1
38.7(1)
7T.8(1)
55.3
2021
37.2(1)
65.5(1)
79.6(1)
41.6(1)
70.9(1)
Luno Ma
37.3(1)
64.6(1)
793(1)
47.4(1)
TT.7(1)
61.3
AT Mia of 20.3
18.1(1) 4.7 61.5(1)
5.8
7T.3(1)
4.1 47.3(1)
6.2
7T.9(1)
6.3
56.4
RMT (Bulatov
37,4(6)
20.4
65.0(8)
79.3(16)
54.6(2)
22.7
61.((4)
12.7
RLT
Kozzchoov et al. 2023
18.4
14.4 61.8(8)
22.6
78.4(16)
12.1 55.0(2)
21.6
74.9(4)
13.6
58.3
RMAAT (Oun))
38.9 38) 5.2
65. 65.9(8)
5.1
83. 83.2(16) 3.4 64.8(2)
5.3
87.1(4)
4.7
68.0
ASc.(S.): Accuracy($)) (Segments used). Mom. (GB): Peak GPU Memory.
These models might have varying sequence lengths in a single segment compared to others. Results are referenced from Tay et al. 2020
" ListOpp (8 R)) length used for segment calculation, resulting segments each with 1024 sequence length.
These results are referenced from puper (Gu et al. (022))
I presents the main accuracy and memory usage results. It compares RMAAT against baselines across the five LRA tasks, showing accuracy per- centages (and segments used for recurrent models) alongside peak GPU memory consumption in GB. RMAAT demonstrates competitive accuracy, particularly on longer context tasks like Retrieval, while maintaining significantly lower memory usage compared to iso-architecture recurrent base- lines. Table details the training throughput. For non-recurrent models (LT, AT), speed is measured relative to the standard Transformer baseline (1x). For recurrent models (RLT, RMAAT), speed is measured relative to the sroatrchiee RMT baseline (1x) to better isolate the impact of the atten- tion mechanism and training algorithm within a recurrent framework. RMAAT exhibits significantly faster training speeds compared to RMT, achieving up to 1.73x speedup on the Retrieval task. This
Performance and Throughput Results: Table

---

Published as a conference paper at ICLR 2026
highlights the efficiency gains from the AMRB training algorithm combined with the ((N) com- plexity of the astromorphic attention framework, compared to RMT's standard BPTT and O(N2) attention. To validate the contributions of RMAAT's core components, we performed several ab- lation studies, primarily focusing on the long-context Byte-Level Document Retrieval (8K) task, supplemented by sensitivity analysis on other tasks (See Appendix (F).
4.2 ABLATION STUDIES
Memory Retention Factor (Con- tributions 1 & 2): Removing
Table 2: Detailed Throughput/Speed Comparison on Long
Range Arena (LRA) Tasks.
the retention factor significantly re- duced aceuracy on the Retrieval task (83.2% + 80.5%) without changing memory usage (3.4 GB), confirming its vital role in context compression derived from the LTP macro model. This effect is consis- tent across modalities: on the Text (4K) task, accuracy decreases from 65.9% 64.9% when the retention factor is removed. AMRB Training (Contribution 3): Replacing AMRB with standard BPTT yielded similar accu- racy but drastically increased peak memory usage on both Retrieval (3.4 GB + 15.0 GB, ~ 4.4x) and Text (5.1 GB - 22.0 GB, 4.3x), demonstrating AMRB's memory efficiency benefits.
Model
ListOps
Text
Retrievalimage Pathfinder
Transformer Vaswani et ml
2017
1x
1x
1x
1x
1x
LT (Kttaropoulos al
1.21x
1.01x
1.03x
1.03x
1.03x
AT (Maa 2025)
1.26x
1.26x
1.05x
1,08x
1.03x
Bulaloo 2022
1x
1x
1x
1x
1x
RIT (Kozachkov dr
1.05x
1.13x
1.37x
1.21x
0.95x
RMAAT (Ours)
1.5x 1.5x 1.73x
1.3x
0.95x
RLT+ + AMRB Ablation (Astromorphic Components): We also evaluate an RLT + AMRB variant, which applies the Memory Retention Factor and AMRB to the Recurrent Linear Transformer (RLT) baseline (standard linearized attention, without Hastro P). On Retrieval (8K), RLT + AMRB attains 79.2% accuracy with peak memory ~ 3.4 GB and RMAAT-like throughput, versus 78.4% and 12.1 GB for RLT and 83.2% and 3.4 GB for RMAAT.
Applicability to Recurrent Architectures: While the Memory Retention Factor and AMRB could potentially offer memory savings if applied to recurrent architectures like RMT (Bulatov et al. 2022). RMT's reliance ( softmax attention creates forward pass bottleneck, limiting throughput gains. Furthermore, RMT would not benefit from the speed improvements associated with the non-linearity and relative positional encoding inherent in the astromorphic attention mech- anism, as reported by Mia et al. (2025).
Total Sequence Length: Evaluating performance on shorter total sequence lengths by reducing the number of segments (while keeping segment sequence length constant at 512) resulted in significant accuracy drops in the Retrieval task (e.g.. 71.5% for 8 segments [4K total length), 65.3% for segments [2K total length] vs. 83.2% for the baseline 16 segments [8K total length)), demonstrating that the model benefits from processing the full context length allowed by the segmentation strategy. Other pyperparameter:: Further analyses in Appendix Fjjnvestiggtee sensitivity to other hyperpa- rameters: spatial range (scale) of the positional encoding and number of memory tokens (M).
CONCLUSION
This work introduced the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), demonstrating an effective approach to efficient long-sequence modeling by integrating computa- tionally abstracted principles from astrocyte function. By incorporating satrocyte-inspiee mecha- nisms for temporal memory compression and resource-efficient training (AMRB)). RMAAT achieves strong average accuracy on the diverse Long Range Arena benchmark. This performance, cou- pled with competitive memory efficiency compared to standard and recurrent baselines (Table [.. validates the potential of leveraging neuro-glial principles for challenging sequence tasks within this benchmark setting. While promising, the current evaluation is primarily focused on LRA; fu- ture work should explore broader domains (including richer vision and multimodal settings), larger model scales, and extensions to true streaming or online processing where the total sequence length is not known in advance, as well as deeper theoretical analysis comparing RMAAT to related se- quence model formalisms. Investigating additional neuro-glial computational mechanisms (such astrocyte communication) and developing specialized hardware implementations also present exciting avenues. In conclusion, RMAAT highlights the value of neuroscience-algon algorrith co-design, suggesting that astromorphic computing is a promising direction for developing power- ful and efficient AI systems capable of handling complex, long-range sequential data.
as astrocyte-astrocyte
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Research was sponsored in part by the Army Research Office and was accomplished under Grant Number W91 INF-24-1-012) The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or im- plied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was also supported partially by the National Science Foundation under award No. EFRI BRAID #2318101.
REFERENCES
Cristina M Alberini, Emmanuel Cruz, Giannina Descalzi, Benjamin Bessières, and Virginia Gao. Astrocyte glycogen and lactate: New insights into learning and memory mechanisms. Glia, 66 (6):1244-122) 1262, 2018.
Malyaban Bal and Abhronil Sengupta. P-spikessm: Harnessing probabilistic spiking state space models for long-range dependency tasks. arXiv preprint arXiv:2406.02923, 2024.
Guillaume Bellec, Francesco Scherr, Elias Hajek, Davor Salaj, Robert Legenstein, and Wolfgang Maass. Biologically inspired alternatives to backpropagation through time for learning recur- rent neural nets. arXiv preprint arXiv:I901.080 2019. URL https:t///sttiii org/ ortppff/ /df/
1901. 09049
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05// 005500 2020.
Brendan A Bicknell and Peter E Latham. Fast and slow synaptic plasticity enables concurrent control and learning. bioRxiv, pp. 2024-09, 2024.
K. Bohmbach, C. Henneberger, and J. Hirrlinger. Astrocytes in memory formation and maintenance. Essays in biochemistry, 2022. doi: 10.1042/EBC2022009
Aydar Bulatov, Yuri Kuratov, and M. Burtsev. Recurrent memory transformer. abs/2207.06881 2022. doi: 10.48550arXi..207..6883 006811..
ArXiv,
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint 1904::1904..000000 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 022660000000,,... 2019.
Maurizio De Pittà, Mati Goldberg, Vladislav Volman, Hugues Berry, and Eshel Ben-Jacob. Gluta- mate regulation of calcium and ip 3 oscillating and pulsating dynamics in astrocytes. Journal of biological physics, 35:383-411, 2009.
Dominique Debanne and Yanis Inglebert. Spike timing-dependent plasticity and memory. Current Opinion in Neurobiology, 80:102707, 2023.
Yu Duan, Zhongfan Jia, Qian Li, Yi Zhong, and Kaisheng Ma. Hebbian and gradient- based plasticity enables robust memory and rapid learning in mns. arXiv preprint arXiv:2302.0323 03335,, 2023.
Marie E Gibbs, Dana Hutchinson, and Leif Hertz. Astrocytic involvement learning and memory consolidation. Neuroscience Biobehavioral Reviews, 32(5):927-944, 2008.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00752. 2023.
11

---

Published as a conference paper at ICLR 2026
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.044: 04451, 2020.
Leo Kozachkov, Ksenia V Kastanenka, and Dmitry Krotov. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120634):c22111501220 2023.
Leo Kozachkov, Jean-Jacque Slotine, and Dmitry Krotov. Neuron- associative memory. Proceedings of the National Academy of Sciences, 122(21))24417788122 2025.
James Teorph Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprini arXiv:2105.088:
2021.
Xuezhe Ma, Xiang Kong, Sinong Wang. Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettle- moyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Sys- tems, 31:::44-22444 2453, 2021.
Qianli Meng. Ming Xiao, Shuang Yan, Yufci Wang, et al. Towards memory-and time- efficient backpropagation for training spiking neural networks. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), pp. 2358-3368, 2023. thecff http:///w...ooooooeeoo/oo///////
URL
htt.://eeee.ffceeeeeeaearrrrraae Spi k iiiiinn_euurrr _ 2t____2222ppperr_____ pdf
Towards @eoward_Memory-i
Memory
for
Md Zesun Ahmed Mia, Malyaban Bal, and Abhronil Sengupta. Delving deeper into astromorphic transformers. IEEE Transactions on Cognitive and Developmental Systems, 2025.
Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O Stanley. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. arXiv preprint 2002::2002.10585.. 2020.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing mas for the transformer arXiv preprint 1rXiv22305.13048 2023.
Hao Peng, Nikolaos Pappas, Dani Yogatama. Roy Schwartz, Noah A Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.021 2021.
Gertrudis Perea, Mriganka Sur, and Alfonso Araque. The tripartite synapse: astrocytes process and control synaptic information. Trends in Neurosciences, 32(8):421 -331, 2009.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-ran range sequence modelling. arXiv preprint ar arXiv:1911.05500 2019.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa-
tions. Xrviv preprint /803.02555 2018
Sergey V Stasenko and Victor B Kazantsev. Model of astrocyte regulation of teikee-iming-deepeee plasticity. In 2023 Fifth International Conference Neurotechnologies and Neurointerfaces (CNN), pp. 86-89. IEEE, 2023.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: successor transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
Yi Tay, Mostafa Dehghani. Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. arXiv preprint 2r 10:20/11.400 1.04006, 2020.
12

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
John Wade, Liam McDaid, Jim Harkin, Vincenzo Crunelli, and JA Scott Kelso. Bidirectional pling between astrocytes and neurons mediates learning and dynamic coordination in the brain: a multiple modeling approach. PloS one, 6(12):e29445 2011.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 2007::0000004768, 2020.
Qingyang Wu, Zhenzhong Lan, Jing Gu, and Zhou Yu. Memformer: The memory-augmented
transformer. ArXiv, abs/2010.06891, 2020.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystrômformer: A nyström- based algorithm for approximating self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14138-14148, 2021.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:23/2.06633 2.06625, 2023.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.
13

---

Published as a conference paper at ICLR 2026
A COMPUTATIONAL NEUROSCIENCE MODEL DETAILS
This appendix provides the detailed equations and parameters for the computational neuroscience model of the neuron-astrocyte network, which forms the foundation for the mechanisms abstracted in RMAAT, as discussed in Section 3.1 The model integrates dynamics across different timescales, capturing key aspects of neuronal activity, synaptic plasticity, and astrocytic modulation.
A.1 NEURAL DYNAMICS
The membrane potential (t) neuron is modeled using Leaky Integrate-and- and-FFre (LIF) dynam- ics. The evolution of the membrane potential is given by:
dv,(t) = -(Vi(t)) - Vreset) + 1((t)
(7)
dt
where:
Tni Neural membrane time constant (Rm C...
(t): Membrane potential of neuron time t.
A: Decay rate for the membrane potential.
Veeset: Reset potential after spike.
4(t): Total input current to neuron i.
When Vi(t) reaches a threshold Vis. the neuron fires spike, and Vt(t) is reset to Vreset- The neuron's ss activity level, Iy, conceptually represents its firing rate or probability, influenced by V,
The imput current (t) is determined by synaptic inputs modulated by synaptic facilitation sty and an intrinsic bias by:
N 1(() = Σ g((r))s,(()) +
(8)
f=1
where:
g(agg): Effective synaptic weight, dependent on synaptic facilitation 8ij- Typically a non- linear function, e.g., sigmoid or linear.
(t): Spike train from presynaptic neuron j. often modeled as ΣΦ 8(t +%) where of spike times.
be Intrinsic bias current for neuron i.
N: Number of presynaptic neurons connected to neuron i.
A.2 SYNAPTIC DYNAMICS
Synaptic facilitation s; between postsynaptic neuron and presynaptic neuron j captures short-term changes in synaptic efficacy. Its dynamics are influenced by neuronal co-activation and astrocyte modulation:
dsij Ts
-ssi + ((x))((x,) v(pi,) caj
dt
where:
Je Synaptic dynamics timescale.
Sty: Synaptic facilitation level between neurons and j.
3: Decay rate of synaptic facilitation.
[z)) Non-linear function representing neuronal activity contribution (e.g., thresholding or sigmoid). 2y are activity levels of neurons i,j.
(pt,)) Contribution from the short-term astrocyte process pi, (modulation). & is typically non-linear function (e.g., sigmoid, tanh).
Short-term astrocyte process parameter associated with synapse (i,
Gyt Baseline bias for synaptic facilitation.
14

---

Published as a conference paper at ICLR 2026
A.3 SHORT-TERM ASTROCYTIC PROCESS DYNAMICS (STP)
The short-term astrocyte process parameter Piy, conceptually related to local intracellular Ca? dynamics near the synapse, evolves based on interactions with other astrocyte processes:
dpig dl
p
Trab(ria + dij
(10)
where:
Timescale for short-term astrocyte dynamics.
Pi Short-term astrocyte process state for synapse (1,
r Decay rate for Ply.
Tow: Coupling tensor representing concentration fluxes spatial influence between the astrocyte process associated with synapse (i.j) and the process associated with synapse 1). It depends the relative spatial distance between these synapses. Specifically, Tyjkı x exp(-distance.j.t x scale). The term distance;j. refers to the Euclidean dis- tance between the spatial midpoint of synapse (ž,j) and the spatial midpoint of synapse 1k,
v (rfa): Non-linear function representing the influence of astrocyte process Pia-
d:y: Baseline bias for the astrocyte process.
A.4 LONG-TERM ASTROCYTIC PROCESS DYNAMICS (LTP)
The long-term astrocyte process parameter ply integrates synaptic activity over longer timescales, contributing to persistent changes and memory:
1 dp's dt
=- + n((ss))
(11)
where:
Timescale for long-term astrocyte dynamics (T/) >> T7).
Long-term astrocyte process state for synapse (i,
r Decay rate for ply-
<(sg)): Non-linear function representing the influence of sustained synaptic facilitation 8ij
on the long-term process.
A.5 PARAMETER VALUES FOR SIMULATION
The specific values used for the simulations presented in the main text (e.g., Figure are listed in Table 3
B ASTROMORPHIC ATTENTION MECHANISM DETAILS
This appendix provides an expanded description of the Astromorphic Attention mechanism em- ployed within RMAAT segments (Section 3.2.2). This efficient mechanism, operating with O(N) complexity, replaces the standard O(N2) sell-attention. Its design fundamentally inspired by computational models of the tripartite synapse, involving interactions between neurons and astro- cytes (Appendix A). and specifically draws from principles of Short-Tera Plasticity (STP) dynam- ics. We conceptualize the mechanism using two-layer network structure (input/hidden and output layers) modulated by astrocyte- like computations (Figure 2}. The process unfolds in two distinct operational phases: Write Mode for context encoding and Read Mode for context retrieval.
15

---

Published as a conference paper at ICLR 2026
Table 3: Computational Neuroscience Model Hyperparameters Used in Simulations.
Parameter
Value
Network Details
Number of presynaptic
Number of postsynaptic ncurons
asimcyles
Simulation Timescale
300
STP Cycle Duration
50 s
Timestep, dt
0.04
Newral Dynamicr
Neural dynamics timescale,
0.5
Membrane potential threshold, Ves.
Reset potential. Vreset
-1 mV
Decay parameterr. X
0.2
Bias parameter,
Non-linearity,
tanh
Synoptic Dynamics
Synaptic dynamics timescale, T
0.75
Decay purameterr, B Bias parameter, Gaj
0.25
Non-linearity,
tanh
Astroeyic STP Dynamies
STP dynamics timescale, Tp
1 0.2
Decay parameter. "
parameter, diy
Non linearity, V
tanh
Astrocytic LTP Dynamicr
LTP dynamics timescale, Tp
6s
Decay parameter, o
0.1
Non-linearity.
sigmoid
B.1 NETWORK STRUCTURE AND INITIAL PROJECTIONS
The core computation is conceptualized within network architecture comprising three functional layers: an input layer, a hidden layer, and an output layer. The input layer receives the segment's combined token representations X E Nd where d is the embedding dimension. This input X consists of Neeq sequence tokens (xxz) concatenated with M persistent memory tokens (mem))) resulting in N N sey M total tokens per segment. The hidden layer consists of m processing units (neurons), acting as an intermediate representation space. The output layer has units, matching the input embedding dimension, producing the final representation for the segment.
Initial processing involves lincar projections of the input X to generate the standard attention com- ponents: Keys (K), Queries (Q). and Values (V). These projections are facilitated by learnable weight matrices that map between the layers:
WK Idxm projects the d-dimensional input X to the m-dimensional hidden space, producing Keys K = XW E Nxm. Keys represent the input signals interpreted or encoded by the hidden layer units (presynaptic neurons).
Wo Rdxm similarly projects X to the hidden space, producing Queries Q XWQ R Queries serve as the signals used later in the Read Mode to probe the encoded context.
Wv Rdxd projects X directly to the output space dimension, producing Values V XWy E Nxd, Values represent the content or features associated with each input token relevant for constructing the output.
Following these projections, non-linear activation function, & (typically d(x) elu(x) 1), is applied element-wise to the Keys (K) and Queries (Q). The resulting o(K) and o(Q) represent the activated states of the hidden layer neurons, signifying their non mon-linear response to the key and query inputs, respectively. These activated states are central to the subsequent Write and Read mode computations.
16

---

Published as a conference paper at ICLR 2026
B.2 WRITE MODE: ENCODING CONTEXT
The Write Mode encodes contextual information from the entire segment by computing effective synaptic weights and abstracted astrocyte state. Conceptually, this involves sequential updates as each token is processed, integrating Hebbian principles with astrocyte-inspired modulation. For effi- cient implementation, these sequential updates are typically realized through final matrix operations performed once per segment.
Neuronal Hebbian Weight nennuon This component represents the direct connection strength between the hidden (presynaptic) and output (postsynaptic) layers, learned via Hebbian plasticity.
Conceptual Per- Token Update: As each token (from N) is processed, its activated key h = ((k, (the t-th row of of () and corresponding value et (the t-th row of V) contribute to the weight update: Hnewron.t = Hneuront- + mh, hig" er (assuming Hnewvon. = 0).
Matrix Implementation: The final weight after processing all N tokens is efficiently calcu- lated as the sum of these outer products:
H Heewvon
Σ -
LKK)VV m
E R
(12)
This captures baseline Hebbian learning, linked to the 0(x,) ((z,))(2,) term (Eq.
Astrocyte-Modulated Hebbian Weight (Hastro): This component models the astrocyte's influ- ence on the hidden-to-outpis connection, incorporating spatial context via a relative positional encoding matrix R E xxm_ The computation of R itself, detailed in Section 3.2.3 and in- spired by STP spatial dynamics (Tiga in Eq. TO), involves transforming a base distance matrix (rij- = exp( -Iloss, pos, x scale)) using learnable projections M and Wret (R = (re((M
Conceptual Per-Token Update: Similar to Hnewran, the astrocyte modulation associated with token t, represented by the t-th row of the activated positional encoding ((I) (let us denote as Φ(r:)), updates the weight: Hostro, Hastro,t-1 + + ((r)) 0t (assuming
Hastro,u = 0).
Matrix Implementation: The final weight is calculated across all tokens:
((r) 1 - _ V Emd
Hastro
(13)
m
This functionally abstracts the astrocyte modulation term (P(t)) (Eq. 9b.
Presynaptic State (g): This vector abstracts the astrocyte's internal state (e.g., calcium level) sponding to cumulative presynaptic activity from the hidden layer. The following two-stage view (linear accumulation of activated keys followed by non-linear transformation on the total sum) aligns with how astrocytes might integrate signals over a period and then exhibit a saturated re- sponse.
Conceptual Per-Token Accumulation: As each token t (from to N) is processed, its ac- tivated key h; ookk) (the 1-th row of ((K)) contributes running sum. If we denote this accumulating sum as Jace, then Jace,t Iacc,t-1 + h₁, starting with Jaee.0 = 0. This represents the linear integration of presynaptic signals before the astrocyte's non-linear re- sponse.
Matrix Implementation and Incorporation of Astrocytic Non-lineariyy : For the entire seg- ment, the total accumulated influence from all N tokens is first computed as the sum MN ((k)). The non-linear saturation effect, modeled by the exponent a, is then applied element-wise to this sum vector (which is of dimension 1 x m) to yield the final presynaptic state g I for the segment:
Σ (kk))
(14)
This g mirrors the temporal integration property of astrocyte processes (p(z) in Appendix A
17

---

Published as a conference paper at ICLR 2026
Combined Hebbian Weight (H): The total effective synaptic strength, H Rmxd is the sum of the neuronal and ustrocyte-modulatee components:
H nevron + H Hastro
(15)
B.3 READ MODE: RETRIEVING CONTEXT
The Read Mode utilizes the activated queries o()) to retrieve the context encoded in the final ag- gregated weights (H) and state (g) computed during the Write Mode. This phase typically involves parallel matrix operations across all N query tokens simultaneously.
Interaction Strength/ Calcium Response (C): Calculates the interaction (C E Nx1 between the current active queries ((Q) and the final presynaptic state g, representing the astrocyte response.
C = =={))
(16)
Fredback Factor (P): Derives feedback factor (P E RNX)),, usually inversely related C, abstracting astrocyte feedback mechanisms.
P 1/C
(17)
Final Attention Output (L): Queries ((Q) retrieve context from H, modulated element-wise (0) by the feedback P. A residual connection adds the original input X. The result L E Nxd is the final output of the attention layer.
L= =
(18)
The expanded form is:
2 ())) + +X
L ==)(Q) (
(19)
(where C' = o(Q) ( (EM o(ke)) ((k)))
This formulation can be compared to standard linearized self-attention, often expressed as SA(X) = ((Q)(((K)))) normalized appropriately. As detailed previously, our equation for L (before the residual connection) shares the core structure ((Q) V), ensuring linear complexity. However, the astromorphic approach introduces two key modifications inspired by the tripartite synapse: (1) The aggregated context includes both direct neuronal correlations (o(K)²) V) and astrocyte- modulated spatial information ()) within H. (2) The retrieved context (H) dynamically modulated element-wise by the feedback factor P = =/CC, which depends on the interaction be- tween the current query Φ(Q) and the aggregated presynaptic state g. This astrocyte-inspiree modu- lation introduces dynamic. context-dependent weighting absent in standard linear attention, while preserving the overall O(N) complexity.
B.4 COMPUTATIONAL COMPLEXITY
The Astromorphic Attention mechanism achieves O(N) complexity per segment with respect the sequence length N, assuming the hidden dimension m and embedding dimension are constants relative to N. A detailed breakdown follows:
Write Mode Complexity Analysis:
Initial Projections (K, Q, V): Calculating K, Q, and V involves matrix multiplica- tions (XWk, XWq. XWv) with complexities O(Nmd), O(Nmd), and O(Nd) respectively. Activation & adds O(Nm).
Hebbian Weights (Hneuron; Hastro): Calculating H Hnenron (6(K)TT involves an m N by N x d multiplication, costing O(N Similarly, calculating Hastro (Ø(R) V), assuming R is computed efficiently, also costs O(Nd)).
- -Presynapic State (g): Summing N vectors of size m (E ((k)) costs O(Nm). Ap- plying the power a costs ((m).
18

---

Published as a conference paper at ICLR 2026
Combined Weight (H): Addition costs ((nd).
Dominant Write Cost: The most significant terms scale linearly with N, dominated by O(Nmd) and O(Nd)) Crucially, the intermediate results H and have dimensions independent of N.
Read Mode Complexity Analysis:
Interaction Strength (C): Calculating C' ( Nxm by mx1 matrix-vector
multiplication, costing O(N)).
Feedback Factor (P): Calculating P (1/C) element- on an N vector,
costing (N).
-Final Output (L): The main computation involves o(Q) @ P). The Hadamard product H @ P requires broadcasting P and costs approximately O(Nmd), if im- plemented by multiplying each row of H by the corresponding element of P. The subsequent multiplication by ((Q) (N m by x d) costs O The residual addition is ((Nd).
Dominant Read Cost: The matrix multiplication dominates, scaling as O(Nmd).
Overall Complexity and Comparison:
Both Write and Read modes dominated by operations scaling linearly with N (primarily (d)).). Therefore, the total complexity segment is O(N).
This linear scaling provides a significant advantage over standard self- attentiennon where the computation the N x N attention score matrix (QKT) leads to an overall com- plexity of D(22d) d). The Astromorphic mechanism avoids this quadratic bottleneck by computing fixed-size intermediate representations (H, g) and using linear-time opera- tions for context retrieval.
C Tige FORMATION AND VISUALIZATION
This appendix details the calculation and visual- ization of the spatial coupling tensor Tijks This tensor is crucial in the Short-Term Astrocytic Pro-
Neuron Positions
Y-coordinate
0.6
cess Dynamics (STP) described Appendix A (Eq. IO.. where it models the distance-dependent in- fluence between different astrocyte processes asso- ciated with synapses (i, and (k:, 1). Understanding its structure helps motivate the bio-inspired relative positional encoding used in Section [.2.3] The spe- cific simulation results visualized in this appendix (e.g., Tija slices and př, dynamies) were generated using a network size of N = 5 and neural bias parameter b = 0.1, run for a duration of 50 sec- onds (representing one STP cycle), with all other model parameters set as detailed in Appendix A (Table 3
0.5
@NO
N1
N2
N3
0.4
0.3
0.0
0.2
0.6
0.8
1.0
X-coordinate
Figure 5: Spatial layout of the NN 5 neu- rons used in the simulation.
C.1 DISTANCE CALCULATION
For simulation purposes, we first define the spatial layout of the neurons. For the example shown, we consider N = neu- rons arranged linearly in a ID space, as- signed coordinates for visualization (see Figure 5). synapse (i,j) connects post- synaptic neuron i. and presynaptic neu- ron j. We define the spatial position synapse (i,j) as the midpoint between the coordinates of neuron and neuron j. This results in grid of N x N 25 possible synapse locations for N = 5 neurons (see Figure 6)
Synapse Positions (Midpoints)
0.8
V-coordinate
(0.4) (2.4 (4-4)
Neurons
0.8
(0.2) (1.2) (2,2)
(0.3) (1.3) (33)
(3.)) (4,2)
Synapoes from NG
Syrapens from
Syrapens from
(0.1) (.1) (2.1) (f1) 66.11
Syrapeax from
10:0) (1.0) (2:0) (3:d) (4:0)
Syropsir
0.4 X-coordinate
1.0
Figure 6: Calculated midpoint positions for all possible synapses between the N = neurons.
19

---

Published as a conference paper at ICLR 2026
C.2 SYNAPSE POSITION CALCULATION
Next, we calculate the pairwise Euclidean distance, distancesy.kt between the midpoint coordi- nates of every pair of synapses (i. and (k, This forms distance matrix capturing the spatial separation between all potential synaptic interaction sites (see Figure
C.3 Tijkt FORMULA AND VISUALIZATION
The coupling tensor Tijki models the strength of influence (c.g., via concentration fluxes like calcium diffusion)) between the astrocyte pro- cess at synapse (i, j) and the process at synapse (k; 1). We model this influence using an expo- nential decay based on the calculated distance:
Pairwise Distances Between Synapses
(4.4)
[4.3]
(.0)
0.8
Synapse Index (f.)
Distance
Tige= diid-dditaneeej.. X scale) (20)
(2.3)
Here, scale is positive parameter that controls the rate of spatial decay. A larger scale value
leads to faster decay, meaning interactions are more localized, while a smaller scale value al- lows for longer-range interactions.
0.2
(0.0)
Visualizing slices of the Tiyki tensor helps to understand the spatial interaction profile from specific source synapse (i. j) to all possible tar- get synapses (k:, l). Figure g shows examples for source synapses located at the corner, edge, grid, for different val- ues of scale. Brighter colors indicate stronger influence (smaller distance or smaller scale). Notice how the spatial extent the influence changes significantly with the scale parameter.
08000r CECEANANNNNN
Synapse
Figure Heatmap visualizing the pairwise Eu- clidean distances between all synapse midpoints for
and center the
i.e., distance.j.ar-
C.4 IMPACT ON ASTROCYTE DYNAMICS AND LINK TO POSITIONAL ENCODING
The formulation of Ti3s as an exponential de- cay of distance (Eq. 220 implies that closer synaptic processes have a stronger potential for direct influence. However, the ultimate astrocytic response, represented by the short-
Spataa Infencction Profile from souree 1i,]1 aceamd3
Tagesthall
term process dynamics pf, (Eq. I0), is not solely determined by Tiju- the complex interplay of neuronal activity Zy), synaptic facilitation (siz), and the in- tegrated influence from all other astrocyte pro- cesses (Ex, (p.)). Therefore, while Tisu defines the strength of individual pair- wise couplings, it is the simulation of the en- tire neuron- astrocyte network that reveals how these distance-dependen couplings translate into spatially modulated astrocytic responses over time.
results from
1978 Taget/ha.
Froe
Togo
Tegilhel i
Figure 8: Visualization of T Tiska slices showing in- teraction strength from different source synapse locations (rows: center, corner, edge) to all target synapses (k./ grid) for varying 'scale' parameters (columns: 2.0, 5.0, 20.0).
As shown in Figure 9 these simulations demonstrate that synapses located centrally in- deed tend to exhibit different temporal dynam- ics for pt (e.g., higher peak and sustained ac- tivity) compared those at corners edges. This occurs because central locations benefit from stronger integrated influence from larger number of relatively closer neighbors, dic-
20

---

Published as a conference paper at ICLR 2026
tated by the itt coupling strengths. Changing the scale parameter in Tiyks further alters the range and strength these interactions, consequently affecting the resulting pi dynamics.
It is this simulated evi- dence-ther the distance- dependent coupling encoded in Tisks, when integrated within the full system dynamics, leads to spatially modulated astro- cytic STP responses (pi;) that provides the biological motiva- tion for incorporating relative positional information in the astromorphic attention mech- anism. Specifically, the base relative positional matrix r, defined in Section 3.2.3 the same exponential decay exp(-distance scale) allow- ing the model to learn suitable spatial interaction range for encoding positional context.
Astrocyte pi Dynamics vs scale
Adtragje Veriable (o.s.)
Certer Smi)
60
(1.0 201
40
C2 Sen2
20
Time
Figure 9: Simulated temporal dynamics of the short-term trocyte process parameter pt; for synapses at different locations (center, corner, edge) under different spatial coupling 'scale' val- ues (2.0, 5.0, 20.0). Simulation uses N = 5, b b= 0.1.
D AMRB ALGORITHM
This appendix provides additional explanatory notes on the Astrocytic Memory Replay Backprop- agation (AMRB) algorithm presented in Section 3.4 (Algorithm []. focusing on gradient flow and implementation details.
Notes:
The indexing convention in the algorithm follows L..I', where m, is the input memory state to segment t, and m4+1 is the output state.
Model( m.) represents the forward pass computation for segment t, producing an inter- mediate memory state m/+1 and segment output Oz.
"m/+1 backaar((raaddenn Vm++1, sraainngraphTTruee. is responsible for backpropagating the gradient from the subsequent segments' losses through the memory pathway of the current segment t.
Explanation of Line 13: Line 13,
Context: During the backward pass, for each segment (from T down to 1), we first compute gradients arising from the local loss I,, of that segment (Line 12). This step, aL, (gradients for model parameters 0, segment and L, backwaaar,,) calculates a0;
also Om,
8Ls (gradient of local loss w.r.t. the input memory mc).
Vm (Upstream Gradient): This is the gradient of the total loss from all future segments (i.c., segments
through T') with respect to m The term m is the memory state that segment t passes to segment f + 1, calculated as m RetentionFactor X m 1- For the first iteration of this loop (when T), Vmr-1 typically initialized to zero the final memory state does not directly contribute to subsequent loss term.
Operation of Line 13: The command barkaabbeccward(grddeennn Vm++1' applies the chain rule. It takes the gradient Vm441 (which is @Lossumm where Loss
L441 + Lt+2 +... + Lr) and computes the gradients of Lossquare with respect to the inputs that formed m Specifically, it computes:
Mossanss by backpropagating Vm4+1 through the operations m4+1 Biss
RetentionFactor x m/41 and m Model( me).
21

---

Published as a conference paper at ICLR 2026
Additional contributions to ossiuu
20, by backpropagating through Model(ay, m,).
The automatic differentiation system handles the scaling by 'RetentionFactor implic- itly when applying the chain rule from m4+1 to m4+1-
Accumulation of Gradients: The gradients with respect model parameters O, and input memory
are accumulated. The gradient Vm, (which will be passed to seg- ment 1 in the next timestep) becomes the sum of the gradient from the local loss (ame
OL2 from Line 12) and the gradient from future losses (oosstum computed in Line
Umi 13). Similarly, parameter gradients V0, also accumulated from both backpropa- gation steps.
retain-ga aetain-grraph = Truc: The computational graph for segment (recomputed in Line 11) is used for two separate backward calls: one for L, (Line 12) and one for m True is necessary for the second call because the first call would typically free the graph. This ensures that intermediate activations and graph structure available for both gradient computations within the current segment t.
(Line 13). sellinngraape
E IMPLEMENTATION AND EXPERIMENTAL DETAILS
This appendix provides supplementary details regarding the experimental setup. hyperparameters, hardware/software environment, and measurement methodologies used for the experiments reported in Section 4
E.1 HYPERPARAMETERS
Key hyperparameters for RMAAT across the evaluated LRA tasks are summarized in Table Con- sistent settings used for iso-architecture baselines where applicable, with task-specific adjust- ments primarily for sequence length handling (Number of Segments) and training schedule (Epochs, Learning Rate). All models were trained from scratch using the Adam W optimizer and CrossEn- tropyLoss where applicable.
Table 4: Key Hyperparameters for RMAAT on LRA Tasks,
Hyperparameters
ListOps (8K)
Text (4K)
Retrieval (8K)
Image (1K)
Pathfinder (1K)
Training Parameters
Batch Size
128
64
16
24
128
Max Len (N)
1024
512
512
512
256
Epochs
100
50
50
100
Learning Rate
5.0e 4
6
Model Architecture
Embedding Dim (d)
256
784
512
784
1024
Number of Heads
2
6
8
6
8
FFN
2048
2048
2048
2048
Number of Encoder Layers
1
1
3
1
Dropout
0.1
0.1
0.1
0.1
0.1
AMRB Recurrence Parameters
Number Segments (g)ng)
Number of Memory Tokens (M)
32
4
32
4
Astromorphic Attention Parameters
Hidden Layer Neuron (m)
100
100
100
100
100
Non-linearity (+)
0.25
0.25
0.25
0.25
0.25
Positional Encoding Parameters Rate spatial decay (scale)
2.0
2.0
2.0
2.0
2.0
E.2 HARDWARE AND SOFTWARE
The experiments were conducted server with the following specifications:
OS: Ubuntu 22.04.5 LTS (Kernel: Linux 6.8.0-52-generic x86_64)
CPU: Intel(R) Xeon(R) Gold 6326 CPU 2.90GHz
RAM:: 503 GiB (approx. 512 GB)
22

---

Published as a conference paper at ICLR 2026
GPU: NVIDIA RTX A5000 (24GB Memory)
Software: Models were implemented using PyTorch version 1.13.1 with CUDA version 11.7. Python version 3.10.13 was used.
E.3 EFFICIENCY MEASUREMENT DETAILS
Peak GPU Memory:
Measured during the training process using standard GPU monitor- ing tools (e.g., nvidia-smi or PyTorch's memory management utilities) to capture the maximum memory allocated the GPU.
Throughput/Speed:: Measured in terms of training time per epoch overall training time, typically reported relative a baseline (e.g.. standard Transformer or RMT). Detailed re- sults are in Table 2 (See Section 4
F ADDITIONAL RESULTS
This section provides additional details and sensitivity analyses complementing the main component ablation results summarized in Section 4.2 The above tables explore the sensitivity of RMAAT's performance (Ac- curacy) variations in the positional encoding spa- tial range parameter (scale) and the number of mem- ory tokens (M) on the ListOps (8K) and Text (4K) tasks. These sensitivity results illustrate the findings mentioned in Section 4.2 performance generally de- grades when deviating significantly from the optimal values for scale and M, confirming the importance of tuning these hyperparameters for each task.
Table 5: RMAAT Accuracy (%) Sensitiv- ity on ListOps (8K).
Parameter
Value
Accuracy (%)
1.0
38.5
scale
5.0
36.8
10.0
36.5
2
37.9
M (Mem Tokens)
8
38.9
16
38.6
32
38.4
Table 6: RMAAT Accuracy (%) Sensitivity Text (4K).
Parameter
Value
Accuracy (%)
2.0
65.9
scale
5.0
65.1
10.0
65.2
16
65.4
(Mem Tokens)
32
65.9
64
65.0
128
65.2
23


Reviews:
[{"review_id": "6SBFy0uAd2", "reviewer": "Reviewer_ZYer", "summary": "This paper proposes RMAAT, a recurrent Transformer architecture that integrates abstracted computational principles from astrocyte-mediated synaptic plasticity. The core ideas include: an $O(N)$ astromorphic attention mechanism inspired by short-term plasticity (STP) dynamics; a Memory Retention Factor derived from simulated long-term plasticity (LTP) dynamics; and a memory-efficient training algorithm that stores only the compact memory tokens across segments and recomputes activations during backpropagation. \n\nThis paper is built upon existing works (including Kozachkov et al., 2023; Mia et al., 2025; Kozachkov et al., 2025). From my own opinion, the core innovation is the memory retention factor derived from simulated long-term plasticity dynamics. This is a good but not groundbreaking paper.  See below.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The neuro-glial inspiration is a fresh angle in the crowded field of efficient transformers.\n- The empirical memory reduction is significant and convincingly demonstrated.", "weaknesses": "- The presentation and writing is not very clear. For one example, how $mem'_{t+1}$ is obtained? \n- Without directly comparing it with current mainstream long sequence models (such as Mamba and RetNet), and only comparing it with traditional Transformer and recursive models, it is impossible to fully demonstrate RMAAT's industry competitiveness in the \"efficiency-accuracy\" trade-off. \n- I still do not know what's the benefit of Astrocyte-inspired model compared with existing methods, such as SSMs, and Linear RNN-Transformers. The stated efficiency-accuracy balance can also be achieved through other solutions. Why not compare? \n- Overstated contributions. Maybe contribution 1 and 2 should be merged into one? ``a Memory Retention Factor from this principle by analyzing the accumulation rate within the simulated LTP curve``. What's the difference between RNN gradient checkpointing and contribution 3? \n- Outdated Baselines: Many baseline results (Sparse Trans., Longformer, etc.) are referenced from the original 2020 LRA paper, which used suboptimal hyperparameters. \n- Weak Ablation Baseline: RLT (Recurrent Linear Transformer) is a weak baseline—it's essentially RMAAT without the retention factor, AMRB, or enhanced positional encoding. A stronger ablation would be RMT + AMRB (to isolate the attention mechanism's impact) or RMAAT without retention factor but with AMRB (to isolate compression's effect).\n- Model Size: The largest model has embedding dim 1024, far smaller than modern LLMs. The memory savings may diminish at scale where activation memory is dwarfed by parameter memory (e.g., 70B models). The paper should discuss extrapolation to larger scales.\n- The connection between AMRB and biological memory replay is tenuous and largely metaphorical, AMRB is structurally identical to gradient checkpointing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YT9BCwW48E", "reviewer": "Reviewer_WnNR", "summary": "This paper introduces a new transformer model inspired by the role of astrocytes in biological neural function. It introduces a novel attention mechanism based on short-term and long-term synaptic plasticity dynamics that reduces the computational complexity of the standard attention mechanism. The model incorporates memory tokens that feed outputs from previous segments of the sequence into subsequent segments, whose influence is modulated by a retention factor calculated from LTP dynamics simulations. Finally, the model harnesses this faster forward-pass computation to implement backpropagation efficiently by recomputing the forward pass from the memory tokens rather than storing the activations. The results of computational neuroscience simulations inspire the values of spatial interaction and temporal compression parameters. With these features, the model achieves state-of-the-art performance on various Long Range Arena benchmark tasks.", "questions": "For the computational macro model (contribution 1), how does the 9-neuron network specifically inform RMAAT’s persistent memory mechanism?\n\nFurther, how does the neuron-astrocyte network specifically inform the architecture, such as the hyperparameters, for the Memory Retention Factor (contribution 2)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The architectural innovations, including the efficient attention mechanism, the use of memory tokens with compression, and the efficient training procedure, are valuable. \n\n2. The benchmark performance results are impressive.", "weaknesses": "1. The explanation of the model’s features could benefit from further refinement for clarity and concision. Figures 1 and 2 should be considerably polished and possibly enlarged. \n\n2. It is not clear how the neuron-astrocyte simulations specifically inform the network architecture beyond guiding priniciples. Further, what happens if one deviates from the values suggested by the simulations? This result might be informative in addition to the ablation analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YfKlsrGn6Y", "reviewer": "Reviewer_dP2F", "summary": "Transformer, especially with attention mechanism, reqjuires quadratic computational complexity which hinders long sequence applications. This paper aims to overcome this limitation by combining astrocyte-inspired mechanisms on transformer architecture, called RMAAT. They also design AMRB alorithm for the update. With proposed method, this paper acheives both state of the art computational efficiency and performance in long range arena tasks.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Biological inspiration from astrocytes combining short-term plasticity and long-term plasticity is interesting.\n- Introducing retention factor for adaptive compression mechanism seems to be novel.\n- Result supports their method is effective in long-range sequence application.", "weaknesses": "**Weakness 1: Insufficient Biological Justification for Core Astrocytic Mechanisms**\n\nWhile I appreciate the ambitious goal of integrating neuroscience-inspired principles to solve the $\\mathcal{O}(N^2)$ problem, I find the **justification for the design of the core astrocytic equations to be insufficient and potentially contradictory** to established micro-scale neuroscience findings. For a model with a vast parameter space, the rigor of biological justification is paramount to ensure the results stem from **methodological insight** rather than an exhaustive hyperparameter search.\n\nFor instance, the dynamic equation for the Short-Term Astrocytic Process Parameter $p_{ij}^s$ (Eq. 2/S4) includes a problematic spatial integration term:\n\n$$ \\sum_{k,l=1}^N T_{ijkl} \\psi(p_{kl}^s) $$\n\nThis term suggests that the astrocytic process at synapse $(i, j)$ is influenced by the **aggregated activity of all other peripheral astrocytic compartments** $p_{kl}^s$ across the entire segment via the distance-dependent coupling tensor $T_{ijkl}$.\n\n*   **Conflict with Microdynamics:** I note that $\\text{Ca}^{2+}$ dynamics in astrocytic peripheral processes—the presumed biological correlate of $p^s$ (Line 650)—are often characterized as **highly localized \"calcium spots\"** (e.g., [1] and related literature), indicating that individual synaptic sites may **not necessarily integrate global** peripheral $\\text{Ca}^{2+}$ signals from distant synapses within the same territory.\n*   **Missing Rationale:** The STP section (Section 3.1) lacks a specific biological or computational neuroscience reference that **explicitly supports the necessity of this broad spatial summation** over $N^2$ synapses. I require a clear articulation of **what specific biological or functional phenomenon** this massive spatial summation is designed to abstract. If it is a macro-scale abstraction of contextual influence, this must be clearly defined and referenced with theoretical literature that supports this abstraction.\n\n**Weakness 2: Lack of Transparency in Core Technical Contribution**\n\nI find the explanation of the **Memory Retention Factor**—one of the paper's key contributions (Contribution 2)—to be **insufficiently transparent and mathematically vague**. This factor, which is derived from the LTP macro model and is critical for adaptive compression (Line 346), is only described conceptually.\n\n*   The exact **mathematical function** of $\\text{RetentionFactor}(t, \\text{TotalSegments})$ is **not provided** in the main text or the relevant Section 3.3, but is relegated to Appendix D.\n*   Given that this factor is claimed to be a **principled, non-learned compression method**, I consider the full mathematical formula to be an **essential part of the core contribution** that must be clearly presented in the main body. The current presentation hinders the model's reproducibility and the community's ability to scrutinize its theoretical grounding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0dpgR8e3z", "reviewer": "Reviewer_9VkK", "summary": "The paper proposes RMAAT, a recurrent long-context Transformer that is explicitly inspired by astrocyte short- and long-term plasticity. Sequences are chunked into segments processed with an astromorphic, linear-time attention block (drawing on STP) and a set of persistent memory tokens that carry state across segments. A non-learned “memory retention factor,” derived from simulations of an LTP-like macro model, adaptively compresses older memory as the number of segments grows. Training uses Astrocytic Memory Replay Backpropagation (AMRB): instead of storing all activations, the model buffers only the sequence of memory tokens and recomputes each segment during backward pass. On Long Range Arena (LRA), the method reports higher average accuracy than a variety of efficient-attention baselines while using substantially less peak GPU memory than iso-recurrent baselines; ablations suggest the retention factor is important for retrieval accuracy and that AMRB yields large memory savings.", "questions": "1. How well does the proposed retention mechanism generalize to settings where the total sequence length is not known in advance, such as streaming or online processing?\n\n2. (Related to the weakness mentioned above) Have the authors explored applying the RMAAT architecture to vision or multimodal data to assess whether the astrocyte-inspired memory dynamics transfer effectively across domains?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a good paper. It is coherent, clear and well-motivated. The biological motivation is nice, and the link to existing literature (in particular the model of Kozachkov, 25) is solid. The division between within-segment linear attention and cross-segment persistent memory tokens is intuitively presented and illustrated clearly. The “retention factor” derived from a long-term potentiation curve is an elegant way to manage bounded memory in long sequences.\n\n The introduction of Astrocytic Memory Replay Backpropagation is also a practical advance: it provides a viable method to train long-context recurrent Transformers under constrained memory without excessive recomputation complexity. The empirical evaluation, while limited in scope, is well-structured within the Long Range Arena framework, and the ablations on the retention factor and memory replay convincingly show their contributions.", "weaknesses": "The experiments are limited to LRA, which is understandable given its focus on long-range dependencies, but it would have strengthened the paper to demonstrate how this architecture performs across other domains—such as vision or multimodal tasks—where long-context modeling may manifest differently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IeM2EqruHX", "reviewer": "Reviewer_RXES", "summary": "The paper introduces a biologically inspired Transformer architecture that models neuron–astrocyte interactions to achieve efficient long-context processing. The proposed RMAAT integrates short-term and long-term plasticity (STP/LTP) into attention, possibly allowing the model to adaptively control how much information is remembered or forgotten over time. Through Astromorphic Attention, attention weights are dynamically adjusted based on astrocyte-like activity traces, while a Memory Retention Factor compresses and consolidates past context. The proposed AMRB method further reduces training memory by replaying compressed states. Experiments on the Long Range Arena benchmark show that RMAAT achieves good accuracy with low memory.", "questions": "1. For the Memory Retention Factor, it seems that the factor is determined based on the total length of the context, but how is this known *a priori*? This appears somewhat strange, especially given that the model is inspired by memory retention in animals, which face a continuous stream of input and must adaptively decide how much information to retain.\n\n2. Hebbian plasticity is described as short-term plasticity in this work, which is somewhat confusing since it is not necessarily considered short-term in a neuroscience context. Is the Hebbian plasticity effectively “cut off” between segments? If the processing time is already linear in the number of tokens, why is it still necessary to divide the context into separate segments? It would also be helpful if the authors could briefly discuss the connection to prior work on neuromodulated Hebbian learning in RNNs (e.g., [1][2]).\n\n3. I noticed that only memory consumption is reported when comparing models. Given the substantially modified attention architecture, I suspect that the forward and backward computation times may also differ significantly from the baselines. It would therefore be helpful to include those comparisons as well.\n\n4. As mentioned in Weakness 3, could the authors expand the ablation study to isolate the contributions of different mechanisms? It seems that only the Retrieval task was used for the experiment removing the Memory Retention Factor; if time permits, it would be valuable to include ablation results across other Long Range Arena tasks.\n\n5. The paper claims the state-of-the-art average accuracy on Long Range Arena benchmark (e.g. line 477), but isn't modern state-space models able to achieve better performance (e.g., [3])?\n\n[1] Backpropamine: Training self-modifying neural networks with differentiable neuromodulated plasticity  [2] Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs\n[3] Simplified State Space Layers for Sequence Modeling", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a novel architecture for long-context modeling by drawing from the biological mechanisms of astrocytes, both short-term and long-term plasticity are incoprated into modern attention-based architecture. It is well-motivated and the links to previous biological models are carefully explained.\n\n2. The AMRB training algorithm reduces the memory burden during model training. The model achieves impressive performance on long-range-arena while maintaining low memory cost.", "weaknesses": "1. While this could be subjective, I find the lengthy method section hard to read. It might be better to directly contrast the vanilla attention with the proposed astrocyte-inspired attention (as the authors did in the appendix), point out which parts correspond to long-term or short-term astrocytic mechanisms, and then explain the link to computational neuroscience models. I also find Figure 2 difficult to interpret, as there are too many interacting components in a small space. For the AMRB algorithm, the pseudo-code in the appendix is much clearer than the textual description in the main section.\n\n2. The model is only evaluated on the Long Range Arena benchmark, which is somewhat limited. For example, including results on language modeling benchmarks (e.g., [1]) could strengthen the paper. \n\n3. The proposed method introduces several interacting mechanisms (Astrocyte-Modulated Hebbian Weight, Feedback Factor calculated with Presynaptic State, Memory Retention Factor), but the effects of these components are not fully disentangled in the ablation study.\n\n[1] The fineweb datasets: Decanting the web for the finest text data at scale, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)