Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections
Authors: Zhuo Liu
Tao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: biologically plausible learning
equilibrium propagation
brain-inspired network structure
residual connection
feedback regulation
Primary area: learning theory
TL;DR: This work proposes a brain-inspired RNN with feedback regulation and residual connections that accelerates Equilibrium Propagation, achieving backpropagation-level performance with enhanced stability and scalability.

Abstract: Brain-like intelligent systems need brain-like learning methods. Equilibrium Propagation (EP) is a biologically plausible learning framework with strong potential for brain-inspired computing hardware. However, existing implementations of EP suffer from instability and prohibitively high computational costs. Inspired by the structure and dynamics of the brain, we propose a biologically plausible Feedback-regulated REsidual recurrent neural network (FRE-RNN) and study its learning performance in EP framework. Feedback regulation enables rapid convergence by attenuating feedback signals and reducing the disturbance of feedback path to feedforward path. The improvement in convergence property reduces the computational cost and training time of EP by orders of magnitude, delivering performance on par with backpropagation (BP) in benchmark tasks. Meanwhile, residual connections with brain-inspired topologies help alleviate the vanishing gradient problem that arises when feedback pathways are weak in deep RNNs. Our approach substantially enhances the applicability and practicality of EP. The techniques developed here also offer guidance to implementing in-situ learning in physical neural networks.

Published as a conference paper at ICLR 2026
TOWARD PRACTICAL EQUILIBRIUM PROPAGATION: BRAIN-INSPIRED RECURRENT NEURAL NETWORK WITH FEEDBACK REGULATION AND RESIDUAL CON- NECTIONS
Zhuo Liu School of Microelectronics
Tao Chen
School of Microelectronies
University of Science and Technology of China Hefei 230026, Anhui, China
University of Science and Technology of China
Hefei 230026, Anhui, China
zhuoliu00@mm
tchen@ustc.edu.cc
ABSTRACT
Brain-like intelligent systems need brain-like learning methods. Equilibrium Propagation (EP) is biologically plausible learning framework with strong poten- tial for brain-inspired computing hardware. However, existing implementations of EP suffer from instability and prohibitively high computational costs. Inspired by the structure and dynamics of the brain, we propose a biologically plausible Feedback Feeuback-regulated REsidual recurrent neural network (FRE-RNN) and study its learning performance in EP framework. Feedback regulation enables rapid con- vergence by attenuating feedback signals and reducing the disturbance of feedback path to feedforward path. The improvement in convergence property reduces the computational cost and training time of EP by orders of magnitude, delivering performance on par with backpropagation (BP) benchmark tasks. Meanwhile, residual connections with brain-inspired topologies help alleviate the vanishing gradient problem that arises when feedback pathways are weak in deep RNNs. Our approach substantially enhances the applicability and practicality EP. The techniques developed here also offer guidance to implementing in-situ learning in physical neural networks.
INTRODUCTION
Backpropagation (BP) has been the driving force behind the success of artificial intelligence (AI) across wide variety of tasks, ranging from image recognition to natural language processing
1986 Lecun 1988, He et al.) 2016 Vaswani et al., 2017). Despite these tri- non local error signals and weight transport lacks biological plausibility Ororbia 2023). The brain does not appear to implement the gradient compu- tations performed by BP, in partieular the explicit derivative of activation function, which demands precise access to the rate of change in neuronal activities at specific operating points (Ororbia 2023) Moreover, implementing BP neuromorphic systems incurs enormous overhead (Kudithipudi al. 2025). Drawing inspiration from the topology and dynamics of the brain is viable approach to ad- vancing biologically plausible learning mechanisms and promoting energy-efficien computing systems for AI.
(Rumelhart et al.
umphs, BP's reliance
(Journé et al. 2023
Equilibrium Propagation (EP) (Scellier Bengio 2017 Ernoult al. 2019
Laborieux al. 2021)
presents compelling and perdware-friendl. alternative. It leverages naturally settling dynamics in RNN for credit assignment, and eliminates the need for explicit activation derivatives. EP operates in two phases with nearly identical dynamics, and the synaptic adjustments depend only local information (Ackley et al. 1985] Movellan [199 Ernoult et al., 2020). In EP, the output layer is softly nudged by the prediction error toward configurations that incrementally minimize the loss function, a regime termed weak supervision (Millidge et al.] 2023). A major drawback of EP is
"Corresponding Author

---

Published as a conference paper at ICLR 2026
its notably slow training speed and instability. An RNN often requires dozens or hundreds of iterations to reach a stable state (Scellier Bengio 2017). Previous attempts to optimize EP's
performance have led to markedly more complicated procedures (0 OOCConnor et al. 2019 Laborieux
& Zenke 2024).
In this paper, draw inspiration from the brain and propose bredback-regulatee REsidual recur- rent neural network (FRE-RNN). We substantially improve the convergence properties of the RNNs and training speed of EP while achieving performance comparable BP. Our contributions are as follows:
By scaling down the feedback strength of RNNs, enhance the robustness of EP and accelerate the training and inference speed by orders of magnitude because the improved convergence properties.
To counteract the gradient vanishing problem caused by weak feedback, we introduce resid- ual connections into the layered RNNs, enabling training deep networks that previously challenged EP and achieving performance closer to BP.
The feedback regulation and residual connections in RNNs of arbitrary graph topologies mirror the multi scale recurrence in biological neural networks. Our work fosters EP's bi- ological plausibility and extend its applicability in brain-inspired computational hardware.
2 BACKGROUND
2.1 CONVERGENT RNNS WITH STATIC INPUT
Consider an RNN as dynamical system driven by static input
(1) where F is the transition function, s(t] is the network state at time step t(t = 0, 1,,,,.... T), and f denotes the parameters. Assuming that the network state stabilizes in T steps, the RNN reaches stable point s(T). Its convergence typically guaranteed by either symmetric connections with asynchronous updates or by sufficiently small spectral radius of asymmetric connections with syn- chronous updates (Hopfield [9822] Yildiz al.) 2012, Liu al. 2026). That said, other factors, e.g. activation function, also influence the dynamical properties of RNNs (Miller & Hardt 2019).
s|t + F(r,
2.2 SCALING ADIACENCY MATRIX TUNE NETWORK DYNAMICS
Scaling the spectral radius (SR) of adjacency matrix, the largest eigenvalue of the weight matrix, is a common method to tune the dynamics of RNN (Bai et al. 2012 Nakajima et al.) 2024 Liu et al.) 2026). A SR less than one yields stable and convergent dynamics. this case, injected signals tend to decay over time, which manifests as short-term memory. A SR exceeding one can give rise to expansive or even chaotic behavior which small perturbations are amplified. By adjusting SR, one can bias the RNN toward convergent, oscillatory, edge-of-chaos regimes, thereby tuning computational properties, such as convergence speed or long-term memory capacity. (Jaeger & Legenstein & Maass 2007 Miller & Hardt 2019)
Haas 2004
2.3 EQUILIBRIUM PROPAGATION
Equilibrium propagation a learning framework initially based on energy-based models. proceeds in two phases: free (first) phase and weakly clamped (second) phase. For the first phase, the RNN converges to a steady state d under the stimulation of input alone. In the clamped phase, the network is gently nudged by the prediction error and settles to new stable state s² The weight update can be simplified to a contrastive learning compatible with spiking time dependent plasticity (STDP) (Scellier al. 2018). EP has been further generalized to asymmetric RNNs governed by vector field dynamics (Scellier et al. 2018). Recent work shows that asymmetry in skew mymmetici Hopfield models can improve classification performance (Høier al. 2024).
2

---

Published as a conference paper at ICLR 2026
2.4 FEEDBACK REGULATION AND NETWORK STRUCTURE IN THE BRAIN
Cortical areas in the brain feature dynamic regulation of feedforward and feedback connections (Felleman & Van Essen 1991 Mejias al. 2016 Michalareas et al.) 2016 Semedo et al.) 2022 Fizek et al. 2023] Wang et al. 2023p. In the visual system, for instance, feedforward signals domi- nate immediately following the onset of external stimulus, whereas feedback signals become promi- nent during spontaneous activity. Dynamically regulating the strength of feedback allows the brain to optimize information integration, ensuring efficient perception and decision-maker malian neocortices, information processing involves not only feedforward synaptic chains but also extensive lateral and feedback loops that interconnect disparate regions, forming a richly recursive network rather than strictly layered structure. This topology implies short average path length between neurons and efficient information flow (Watts & Strogatz 1998 Markov et al. [2013] Lynn & Bassett 2019: Kulkarni & Bassett 2025). In deep neural networks, residual connections reflect the long-rarn range skip-layer projections observed in cortical circuits (Perich Rajan 2020) Holk &
In mam-
Mejias 2024). They mitigate vanishing gradient by providing skip pathways that preserve gradient
(He et al. 2016
3 ACCELERATING EP WITH BRAIN-INSPIRED NETWORK PROPERTIES
(a)
51
$2
Predict: Sp
O
Wo
200
01
W
0000
Label:
B
B
Error:ep 5. Sp
(b)
51
$2
Predict: Sp
Conny
Ps Convs
P2. W/
Label: St
000
(325..00
2) (64.5.00
(2)
Erroriep S; Sp
Figure 1: Illustration of feedback and feedforward regulation. (a) Layered architecture of RNN. The feedforward weights W, and feedback weights B, are rescaled by coefficients a; and 3, respectively. The dashed box encloses an RNN formed by layers $1 and $2 with feedforward and feedback path- ways. By is the nudging factor, which essentially scales the feedback strength of prediction error. (b) Embedding convolutional architecture in RNN. Convolutional parameter (32,5,1,0) is written (channels, kernels, stride, padding). Parameter (2) in (b) denotes max-pooling with stride 2. Conel; represents transpose convolution, the inverse process of the convolution, and P.". means max-unpooling (Ernoult et al. 2019). Model architectures and training process are given in Ap-
pendix D
3.1 PROTOTYPICAL SETTING OF EQUILIBRIUM PROPAGATION
Unlike the prototypical setting of equilibrium propagation (P-EP) Ernoult et al. 2019). we separate the input and output layer from the recurrent network (Figure TT). This separation allows the output layer to adopt the SoftMax activation commonly used in feedforward networks, which facilitates performance comparison (Laborieux Zenke, 2024). For clarity, the RNN (black dashed box in Figure ( shown here only contains two hidden layers and 82, but the approach applies to deeper structures (see below). The states of the RNN evolve for T' discrete steps until they converge. The dynamics of the whole RNN can be formulated as:
8 [t + 1] = F( 9 [t], b) = p(W 8 + b).
b [Wo So.
By By epl,
(2)
3

---

Published as a conference paper at ICLR 2026
where s²1 [t] is the state of the RNN at time is the activation function, W is the forward weight matrix of the RNN, and b combines the feedforward input and the error-nudging term. Note that
8os = [ $2 I. For each sample-label pair (æ, Star), we run the free phase (3, = 0) for to iterations, obtain the prediction 8p = SoftMax (Wy. 82), and compute the prediction error ep Star During the clamped phase, the error nudges the RNN through the feedback weights By and scaling coefficient B, = f (6 = 0.1 for layered architecture and f = 0.25 for convolutional architecture by default). The network evolves for K further iterations under clamping to another state. The weights (Wo, Wi) are then updated with an STDP-compatibl rule:
AW; = dss+1 (s))T,
d8 = s,41 S.+1
(3)
where ds; is the offset of stable point caused by the error nudging (Scellier et al. 2018). Similarly, the final weight for output is updated:
AW, = (star - sp) (s2)
(4)
We also consider an RNN embedded with convolutional architecture in its forward paths (2 convo- lution layers, 2 max-pooling layers and fully connected layer) shown in Figure Mp. The forward convolutional structure follows the architecture of existing convolutional neural networks (CNN) (Krizhevsky et al.,, 2012; [Smonyynn Zisserman, 2015), which pooling layer is placed after the activation of the convolution layer. We transform the CNN to an RNN by adding feedback con- nections symmetric with the feed-forward connections (See Appendix D for the pseudocode and schematics).
3.2 FREDBACK REGULATION LAYERED RNN FOR FAST CONVERGENCE
(b)
Inbeer
G
300
100
10-2
bodet
10-4
(
100
100
10a
300
100
20
60
80
20
40 60 00
Figure 2: Convergence dynamics and speed versus feedback scaling 3,.. All neurons in all hidden layers are indexed (s) (sy:0-63; s2:64-127). Colors indicate neuronal activity (a,c.e.g) and changes in activity (b,d,f.h). (a) The state evolution of RNN with symmetric weights and B = 0.1; (b) The one-step difference of neural states in (a). (c. d) Symmetric weights with 8; = 2; (e, f) Asymmetric weights with 3, = 0.1; (g. h) Asymmetric weights with 3, = 4. In both symmetric and asymmetric feedback cases, down-scaling feedback connections tends to stabilize the network. See Figure 5l for the statistical robustness.
Although the SR can tune the RNN dynamics, scaling forward weights W, distorts forward signal propagation, which is harmful to performance (see below). Therefore, we turn to another choice, namely, scaling only the feedback strength with Sg. This coefficient scales the gradients, in the same way as the nudging factor 3y. We consider both symmetric (B, (Wt) T) and asymmetric (B, * TW recurrent connections in the study, and compare the results with FNNs of the same size trained by BP (feedback connections removed) or feedback alignment (FA) (Lillicrap et al., 2016) that uses random weights B, * (W;) to feedback the error. Note that, after scaling, the overall weight matrix W of a symmetric RNN is no longer strictly symmetric. Therefore, we started from the vector field setting of EP rather then the energy-based setting in the first place. The feedforward and feedback weights are multiplied by coefficients a; and 3 respectively. Figure 2 shows convergence speed for different 3,. With asymmetric weights, the network can converge point (Figure 22. f), exhibit cyclical oscillation (Figure 23g h), or even become chaos. The feedback weights stay fixed during training process, which differs from EP in vector field dynamics (Scellier
fixed
4

---

Published as a conference paper at ICLR 2026
The pseudocode of learning procedure with --hhdden-llyee RNN shown in Figure a)) is provided in Algorithm
al. 2018).
Algorithm EP with Feedforward and Feedback Scaling
Require: Input: (xr Stte))
Require: Parameters: 0 = [Wo. Wi, W,, By, an, 81, 8n]
Ensure: Updated parameters 0
1: function FIRST-PHASE(O, Star)
So f- x:
3: for = to T do
4:
h1 - Wo + 51 Wi .s
Bı -2
5:
h2
6:
hpp W -22
$1, 02 0 - p(h1), p (ha), Soft
8: end for
9: A1 [s 0, 1,2
10: return A
11: end function
12: function SECOND SECOND-PHASE Star)
Star)
13:
81 811 841
53 :8p
+ 80 82, 8p
14: for = 1 to K do
8y1
15:
Ep Star Ap
16:
h Wo
B1 B1 82
8r
17:
h2 an
W1
81
8t
+ Bg By Cp
hp < W b2
P
19:
82 5p
8n
8n1
+ p(h1), p(22), (ofth)
20:
end for
21: ds; 8n 8 1,2
22: A2 1ds ds2
return A2
24:
end function
25:
function UPDATING- WEIGHITG- A1, A2, Star)
26:
AW, dsa41 (8 i 0,1
27: AW, 4- (Star sp) (89)
28: end function
3.3 RESIDUAL CONNECTIONS TO AVOID VANISHIG GRADIENTS
In 10-hidden-layer RNN with symmetric connections, add cross layer residual links (Fig- ure Bh-b) and carry out ablation study on their effects in performance. The three long-range bidi- rectional connections bypass adjacent layers reduce gradient decay. For RNN with asymmet- ric connections, we introduce skip-layer connections between non-adjacent layers with / = 20% probability, creating an RNN with arbitrary graph topologies (AGT) where any pair of layers form connections stochastically (FigureBb) (Salvatori et al. (022). (See Algorithm (S Appendix for training detail)
EXPERIMENTS
We evaluated our RNN models MNIST and CIFAR-10 datasets and compared the results with P- EP and BP. The MNIST dataset consists 70,000 grayscale handwritten digit images (28x28 pixels) split into 60,000 training and 10,000 test samples. CIFAR-10 contains 60,000 RGB images (32x32 pixels) categories, divided into 50,000 training and 10,000 test samples. Pre-processing, net- work structures and additional training details are in the Appendix D
4.1 INFLUENCE OF FREDFORWARD SCALING AND FEEDBACK SCALING
Figure 4 compares the effects of feedforward scaling a; and feedback scaling 3,. In general, relative small feedback scaling (3: 0.1) yields high MNIST accuracy (Figure 4). In deeper RNNs, overly

---

Published as a conference paper at ICLR 2026
(a)
HT0
Out
S
Layer out
Figure 3: (a) A 10-hidden-layer RNN model with residual connections. The solid blue wires and the dashed orange wires represent forward and feedback residual connections respectively. The bidirectional connections are symmetric. (b) Adjacency matrix of (a). The blocks (green) other than the sub-diagonals indicate residual connections. (c) Adjacency matrix for an RNN with arbitrary graph topology.
2HL--est accuracy
3HLL-lest accuracy
SHL-Test accuracy
0.001
8.001
0.001
9000 6 9723 0 3760 0. 5758
0.01
0.226
3.5575
5.5 35155
0,01
9.2515
039355
85555
0, 0.8583
0.85
0.80
1.0
0.8925 0. 9127
0.9300
0. 7978
1.0
0.4952
0.8739
0.5249
0.1676
1.0
0.2096 0. 0.3891 0,2500
0.1324
0.75
0.01
0.1
1.0
4.0
0.01
0.1
1.0
4.0
0.02
0.1
1.0
Figure 4: The influence of feedforward scaling and feedback scaling 3, on accuracy of MNIST classification. (a) 2 hidden layers; (b) 3 hidden layers; (c) hidden layers. Each layer has 64 neurons. By default, T 10 x Ndddden K T/2. Each result is averaged over five repetitive experiments.
(a)
(b)
(c)
(4)
101
102
Corvergence
10:3
104
10*0
120
0.01
6,003
0.1
0.803
0.0
000
Bal
Figure 5: The test error, SR, FTMLE, and convergence time versus with feedback scaling 3, The results are obtained from a 3-hidden-layer (64 nerons per layer) model trained on MNIST dataset. Note that the network does not converge under certain conditions, resulting in missing value in d.

---

Published as a conference paper at ICLR 2026
low feedback scaling 3; jeopardizes the performance, which we attribute vanishing gradients (Figure De. right two columns). In contrast, down-scaling the feedforward weights degrade perfor- mance, as the inference signals weakened through the layers (see rows of Figure However, up-scaling or can also be detrimental, as this easily leads to saturation of neural state. Thee best per- formance of 5-hidden-layer RNN is achieved without feedforward scaling 0; and trade-off in feedback scaling at B, 0.1. These results suggest that balancing the feedforward and feedback strengths is critical for better performance, not only accuracy but also speed (see Table 1).
To further investigate the influence of feedback scaling 3,, we plot the error, SR, finite time max- imum Lyapunov exponent (FTMLE) (Shadden et al., 2005| Kanno & Uchida 2014) and conver- gence time against feedback scaling coefficient before and after training of a 3-hidden-layee RNN on MNIST (Figure 5). shows that larger feedback scaling 99 decreases accuracy (Figure 5p. As expected, SR is positively correlated to A3 (see Figure 5). and large SR can lead to instability of an RNN indicated by the FTMLE shown Figure 55. which in turn explain the results in Figure 5a. In general, down-scaling the feedback (8; < 1) reduces the convergence time of RNN, which is favorable. Note that up-scaling of feedback 3; > can also decrease FTMLE and convergence time. However, this is attributed to the saturation of neural state, and will also lower the performance.
Additionally, one might suspect that the gradient signals in the lower layers are not fulfilling their intended role. In reservoir computing, where only the last layer is trained, the network can also reach high accuracy as long as the output dimension is large enough. However, this is unlikely in case, as each layer in our network has only 64 neurons by default (other than the results in Table 1). To further confirm that the learning in lower layers are meaningful, we performed training with of these experiments are included in the Appendix C.5 The results clearly show that the getting comparable results to BP requires effective training of lower layers.
the weights of lower layers
32
R-0.01
[b)
10
A-0.1
5-10
10'
A-4.0
20-4
23
Tustn
20
100
30
50
1.1
33
10
10
20
Fpwr
kparh
Layers
Layers
Layers
10 Layerss
32"
M
Teare
21-
5-22
22
20-
E003
0.002
2.3
625
50
33-1
10
40
Speci
Opach
Spach
Figure 6: Test error with different hyperparameters. The curves of different T (10, 20, 50, 100) with
2 hidden layers (64 neurons per hidden layer) and (a) 8, 0.01; (b) 3 = 0.1: (c) 8 1; (d) 3, = 4. The curves of different 3; (0.001, 0.01, 0.1, 0.25, 1, 2, 4) with (c) 2 hidden layers; (f)
hidden layers; (g) 5 hidden layers; (h) 10 hidden layers. The shaded areas represent deviations of
five repeated experiments. By default, T = 10 x NiiddenLager, K = T/2. See Appendix A for more information.
4.2 DOWN-SCALING FEEDBACK LEADS TO FASTER CONVERGENCE
Figure o plots the error versus the number of epochs with different iteration steps T. Under the condition of 3, 0.01 (Figure 6a), the model with T 10 and K 5 works well as the model with T = 100 and K = 50, suggesting possibility of speedup in training. Larger 3; requires more iterations achieve certain level of performance (See Figure tp. c, d). Larger (3, means larger SR
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with P-EP and BP in accuracy and cost. The results of P-EP come from previous work (Ernoult et al. 2019). For BP results, we used network with the same number of layers and number nodes/channels. Each experiment repeated five times, and the standard deviation is given. By default, 3 0.01 in our results, the feedback weights are symmetric with the feedforward weights for P-EP and Ours, and the learning rate in layers are the same except for Ours-DLR (different learning rate), which uses varying learning rates identical to that of P-EP. For 2HL (two hidden layers) and 3HL (three hidden layers), there are 512 nodes per hidden layer. See Appendix D for more details.
Epoch Batch size
Wall Clock Time (HH::M:SS))
Architecture
Training approach
Testing (Training)
-T/K
P-EP (sigmoid-s)
98.05%10.100%
(99.86%)
50/20-100/20
1:56:-
2HI.
Ours (tanh, Adam)
98.39%%+00.0% (100.00%)
50/500-10/10
0:01:16
BP (tanh, Adam)
1..06%% (100.00%)
50/500-1/1
0:00:18
P-EP (sigmoid-s)
97.99% 18% (99.90%)
8:27:-
Ours-DLR (tanh)
97.65% + +0.08% (98.93%)
100/20- 18/10
1:01:14
3HI.
Ours (tanh)
97.83% 13% (99.98%)
100/20-18/1 18/10
1:01:54
Ours (tanh, Adam)
98.36%:10006% (100.00%)
50/500-1880 18/10
0:02:11
(tanh, Adam)
98.36% 08%% (100.00%)
50/500-1/
0:00:24
P-EP (hard-sigmoid)
98.98%+000444
(99.46%)
40/20-200/10
8:58:-
Conv
Ours (hard-sigmoid)
99.14% +0.0%% (99.78%)
40/128-20/10
0:12:28
BP (hard-sigmoid)
98.93% +0..1 18% (99.43%)
40/128-1/1
0:01:01
Table 2: Comparison with BP and FA and ablation study of residual connection. For layered architecture, there are 64 nodes per hidden layer and we chose T = 10 x N HiddeenLyerr and K = 5 x NHidden Layers which guarantees saturation of accuracy at 8 0.1. For convolutional architectures, 3, = 0.01. By default, the Adam optimizer used. Each experiment is repeated five times. See Appendix more training details. Architecture
Training approach MNIST-Testing (Training)
CIFAR-10-Testing (Training)
-conecctonns
BP
97.69%++0.105 10%% (100.00%)
49.23% (56.72%)
5-symm
Ours
97.64%:10.100 10% (99.98%)
50.72%:10.17% (57.02%)
FA
96.44%10.1099 10% (98.96%)
37.97%=2.18%% (38.92%)
5-asymm
Ours
96.37%10.111 11% (97.99%)
45.27% (46.79%)
BP
97.61%+t0.04% (99.93%)
48.23% 1.66% (55.37%)
10-symm
Ours
92.49%++..32% (95.27%)
34.90%:1-0388% (34.64%)
Ours-Residual
97.49% (99.77%)
44.46% % (48.67%)
FA
94.52%:10.26% (95.54%)
30.16%11.. 12% (30.20%)
10-asymm
Ours
87.37%:10.49% (87.95%)
30.37%:+1.09%% (29.97%)
Ours-AGT
96.87%+0..1 11%% (99.45%)
30.94%+14.90% (31.36%)
BP
97.48% (99. 74%%
17.35%+1.4999 (54.59%)
20-symm
Murs-Residual
95.95%+1. 18% (98.)0%)
43.61% (44.26%)
BP
99.34%++0.04% (99.97%)
75.45%:100446% (83.61%)
Cony
Ours
99.27% (9%)78%)
75.04% % (80.79%)
and FTMLE, thus requiring more iterations to settle the RNN as shown in Figure and Figure 5 d). Or even worse, the gradient signal is completely distorted. At 8, = 4, even
T T=100 fails exceed 95% accuracy. Figure 6 shows that while shallow networks benefit from low 8, deeper networks (3, and 10 layers) lose accuracy. In all cases, training performance peaks at certain 3, dependent on the network depth. Additional results provided in Table ST in Appendix B
Table compares our approach with P-EP, BP, and FA. Our model supersedes P-EP in training speed by least one order of magnitude for both convolutional architecture and layered architecture. Importantly, our accuracy is comparable to BP and FA for the shallow architectures (5-hidden-layer and conv model, also Table 2g. In consideration of the improved stability (Figure 6) via feedback
8

---

in Ap- Published as a conference paper at ICLR 2026
regulation, anticipate that physical implementations of RNN achieve performance par with BP. Additionally, for layered architecture, we also adopt the same training parameters (learning rate, batch size and epochs) P-EP, differing only in feedback scaling ('ours-DLR in Table The results present clear evidence of speedup, which mainly stems from the reduced number of iterations required for convergence.
4.3 DOWN-SCALED FEEDBACK COORDINATES PLASTICITY OF DIFFERENT LAYERS
It is hypothesized that the brain requires different plasticity in different areas due their varying functional roles (Atallah et al. 2004 Lowet et al. 2020). The variability in plasticity can be realized explicitly by adjusting learning rates implicitly by modulating intensity of gradient. Previous work postulated that EP with weak feedback necessitates learning rates differing by orders of mag- nitude across layers (Scellier & Bengio 2017). Here, we found that due to gradient differences across different layers induced by weak feedback, a 3-hidden-layer RNN at B; = 0.01 (Table II "ours (tanh)") learns well with uniform learning rate. This result suggests that the feedback scaling alone is able to regulate gradient strength of different layers, pointing to another possible mechanism coordinate plasticity.
4.4 RESIDUAL CONNECTIONS OVERCOME THE GRADIENT VANISHING DEEP RNNs
Weak feedback exacerbates vanishing gradient in deeper layered RNN (Figures S5 S6
pendix [B. Adding residual connections restores gradient flow (Figure S7 in Appendix B As a result, a 10-hidden-layer network sees substantial performance gains (Table 2 5% increase in ac- curacy for MNIST and 9% for CIFAR-10, 10. Even 20-hidden-layer model can be trained. As shown in Table 2 without residual connections, an asymmetric RNN trained by EP falls short of FA in accuracy, but arbitrary residual links surpass the accuracy of FA for the MNIST classification (See ablation study on connection probability in Appendix B )- For more complex dataset CIFAR-10, the 10- 10-hidden-layer asymmetric model with residual random feedback connections achieve accu- racy nearly 14% below the symmetric model. A possible reason is that the gradient signal through
multiple random fixed feedback connections become distorted by error coordinate the forward weight learning.
DISCUSSION
We have applied the feedback scaling to RNN to speed up the convergence and to accelerate training with EP with negligible overhead. To counteract the vanishing gradient in deep architecture, we have added residual connections to non-adjacent layers of deep RNNs, partly restoring classification performance. In principle, the residual connections makes credit assignment pathways shorter (Veit) et al. 2016). The training exhibit remarkable resilience to noise weight and neural state. Our structural modification is compatible with other algorithmic speed-ups (Scellier et al.) (023). thereby expanding the design space for efficient EP implementations.
Recent work on credit assignment in brain-inspired networks, e.g. adjoint propagation (Liu et al., 2026), partitions large network into local RNNs with random internal connections of low SR For fast convergence and dynamic resource allocation, yielding speed and accuracy similar to this work. This work, however, adopts the feedback scaling to solve the stability issue and accelerate convergence speed of EP.
Weak feedback is often considered in biologically plausible learning algorithms (Sacramento al., 2018 Haider et al.] 2021] Meulemans et al.] (021). It has been shown that contrastive Hebbian learning with weak feedback approximates hack-propag while converging quickly (Xie & Se- ung] 2003). More recently. local representation alignment (LRA) )) likewise employed weak feedback OOrorbia et al., 2023) and skip connections from the output to deep layers for efficient training. The
EP framework also approximates BP (Scellier & Bengio 2017; Millidge et al.] 2023), but under the weak clamping condition (weak supervision) (Laborieux et al. 2021 Millidge et al. 2023). We have shown that, at the infinitesimal inference limit, namely weak supervision and weak feedback (Milldgee et al. 2023), EP is equivalent to LRA and BP (Appendix (C). In other words, the dynamics of FRE-RNN more like to the feedforward neural network due to its weak feedback.

---

Published as a conference paper at ICLR 2026
However, there still few limitations to our approaches for large- neural network that under- pin artificial intelligence. For complex datasets like CIFAR-10, there exists a notable performance gap compared to BP, using deep fully connected neural network. We attribute this gap to the inaccu- rate approximation to the true gradient as computed by BP (See Appendix (.4). Therefore, although EP can be extended to deep fully connected network (20-hidden-layers) and shallow CNNs, its appli- cability for deep CNN remains to be explored. For deep architectures with asymmetric connections, the accuracy decreases faster with increasing depth due to the inaccurate random error feedback. More in-depth investigation on residual connection topology is required to scale up the methodol- ogy large scale deep architectures. Besides, the hyperparameters are optimized empirically. We find a feedback scaling in the range of 0.01-0.1 is favorable for shallow networks (less than lay- ers) and 0. 0-1.2. for deeper architectures. Finding a general way to determine these parameters is still on-going. Additionally, existing EP converging naturally research continues to focus primarily on static-input settings (Laborieux et al. 2021 Ernoult et al., 2019 Laborieux & Zenke 2024 Extending naturally converging RNN trained by EP to sequence tasks remains challenge.
From a neurobiological perspective, residual connections, particularly the randomly generated arbi- trary graph topologies, yield cortex-like connectivity patterns in the brain. The geedback-regulann residual RNNs equip the biologically plausible learning framework, EP, with biologically plausible network architecture. Although it currently runs on GPUs, it exploit the natural convergence of physical RNNs and facilitate efficient learning and inference on dedicated neuromorphic hardware.
ACKNOWLEDGEMENTS
We acknowledge financial support from the University of Science and Technology of China and the Chinese Academy of Sciences.
CODE AVAILABILITY
The code used in this work is available httpss//ggttubb.comm/eerrttf
FRE RNN EP)
REPRODUCIBILITY STATEMENT
The code necessary to reproduce the main results is provided Jupyter Notehooks in the Supple- mentary Materials. Researchers can directly run them to reproduce the results. Further details on data pre-processing and training process are available within the provided code and in Appendix D
THE USE LARGE LANGUAGE MODELS (LLMs)
In the preparation of this work, the authors used GPT-5 and Deepseek solely for the purpose of polishing and improving the linguistic fluency and readability of the text. This includes tasks such as correcting grammar and rephrasing sentences, After using the model, the authors have reviewed and edited all content extensively and take full responsibility for all ideas, claims, and the final language presented in this paper.
REFERENCES
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltz- mann machines. Cognitive Science, 91)::177-66) 1985.
Hisham E. Atallah, Michael J. Frank, and Randall C. O'Reilly. Hippocampus, cortex, and basal
ganglia: Insights from computational models of complementary learning systems. Neurobi-
ology of Learning and Memory, 82(3):253-265 2004. ISSN 1074-7427. doi: https://doi. 11111..116/..nmmnn www.://.............. .0004. URL https: : wwwp www.ssciiieencc.e..ccecc com/sciennes Atticcle/p11/11074422222
Zhang Bai, D. J. Miller, and Wang Yue. Nonlinear system modeling with random matrices: Echo state networks revisited. IEEE Transactions on Neural Networks and Learning Systems, 23(1):
175-182, 2012. ISSN 2162-237X 2162-2388. doi: 10.1 20tp:/mnns.2111111111
10

---

Published as a conference paper at ICLR 2026
Jimmy Ba Diederik P. Kingma. Adam: A method for stochastic optimization. In International Conference for Learning Representations. ICLR, 2015. doi: dbi.org/10.485550aara3 www...111.......
Maxence Ernoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, and Benjamin Scellier. Updates of equilibrium prop match gradients of backprop through time in an mnn with static input. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. Article 636. Curran Associates Inc., 2019.
Maxence Emoull, Julie Grollier, Damien Querlioz, Yoshua Bengio, and Benjamin Scellier. Equilib- rium propagation with continual weight updates, 2020. URL httph:tt//////tt http://ppveevvveeeeeeeeeee eww net.
D. J. Felleman and D. C. Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral Corlex, 1(1):1477, 1991. ISSN 1047-3211. doi: 10.1093/cercor/1... URL
<GotoIsI>: ://10///W05> 000000047200000000
Mehmet Fişek, Dustin Herrmann, Alexander
Matilda Cloves, Lisa Bauer, Tai- Ying Lee, Lloyd E. Russell, and Michael Häusser. Cortico-cortical feedback engages active den- drites in visual cortex. Nature, (17777662)77997766 2023. ISSN 1476-4688 doi: 10. 1031038/ 941586-023-06007-6 URLhttps: wtwp:/ow.......ooo 1038/ 441886 023 06007-6
Paul Haider, Benjamin Ellenberger, Laura Kriener, Jakob Jordan, Walter Senn, and Mihai A. Petro- vici. Latent equilibrium: a unified learning theory for arbitrarily fast computation with arbitrarily slow neurons. In Proceedings of the 35th International Conference on Neural Information Pro- cessing Systems, pp. Article 1365. Curran Associates Inc., 2021.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. ISBN 1063-6919. doi: 10. 10.100CCCRR..20...0.
Maya van Holk and Jorge F Mejias. Biologically plausible models of cognitive flexibility: merging recurrent neural networks with full- full-brain dynamics. Current Opinion in Behavioral Sciences, 56: 101351, 2024. doi: 10. (/16///.cob/ettt... 20..2.224...3. van Holk, Maya; Mejias, Jorge F. ORCIDs: 0000-0003-4930- 49900-4---9--22222 2352- 1546 2352-1554.
J. J. Hopfield. Neural networks and physical systems with emergent collective computational abili- ties. Proceedings the National Academy of Sciences, 79(8):2554 2558,, 1982. ISSN 0027-8424 1091-6490. doi: 11.1073/pnas.79..2255
Rasmus Høier, Kirill Kalinin, Maxence Emoult, and Christopher Zach. Dyadic learning in recur- rent and feedforward models. NeurlPS 2024 Workshop Machine Learning with new Compute Paradigms, 2024. URL bettps://openrevveee http://ppppreee....eeeee net httr://rrrnmn--LLNW/oeee
Herbert Jaeger and Harald Haas. Harnessing nonlinearity: Predicting chaotic systems and sav- ing energy in wireless communication. Science, 304(5667))788800, 2004. doi: 10.1 10.11/6//cience. 1091277. URL ence. 1091277.
doi: 10.1
Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learn- ing without feedback. The Eleventh International Conference on Learning Representations, 2023. htttltttt://oeeeett
ew, net. forum? 3
Kazutaka Kanno and Atsushi Uchida. Finite-time Iyapumov exponents in time-delayed nonlinear dynamical systems. Physical Review E, 89(3):032918, 2014. ISSN 1539-3755 1550-2376. doi: 10. 10.11033Phy.Reev.....3333
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, 2012.
Dhireesha Kudithipudi, Catherine Schuman, Craig M. Vineyard, Tej Pandit, Cory Merkel, Rajkumar Kubendran, James B. Aimone, Garrick Orchard, Christian Mayr, Ryad Benosman, Joe Hays, Cliff Young, Chiara Bartolozzi, Amitava Majumdar, Suma George Cardwell, Melika Payvand, Sonia Buckley, Shruti Kulkarni, Hector A. Gonzalez, Gert Cauwenberghs, Chetan Singh Thakur, Anand Subramoney, and Steve Furber. Neuromorphic computing at scale. Nature, 637(8047):801-88 2025. ISSN 0028-0836 1476-4687. doi: 10. 1..1008//411 586-024-08853--8

---

Published as a conference paper at ICLR 2026
Suman Kulkarni and Dani S. Bassett. Toward principles of brain network organiza- tion and function. Anmual Review of Biophysics, 54(Volume 54, 2025):353-378, 2025. ISSN 1936-1238. doi: URL
http://bwwwooolloaoomouuuubbbbba 110624. http:::/www.aaatt annua I http://wwww....ee...o.ooo.///ota
Axel Laborieux and Friedemann Zenke. Improving equilibrium propagation without weight sym- metry through jacobian homeostasis. In The Twelfth International Conference on Learning Rep- resentations. ICLR, 2024. http:tss: sppenreiieeeeee net net/
Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation deep convnets by drastically reducing its gradient estimator bias. Frontiers in Neuroscience, 15:633674, 2021. ISSN 1662-453X. doi: 10.3389/ Nnins.2021.63361
Yann Lecun. A theoretical framework for paok-propagation. Proceedings of the 1988 Connec- tionist Models Summer School, 1988.
Robert Legenstein and Wolfgang Maass. Edge of chaos and prediction of computational perfor- mance for neural circuit models. Neural Networks, 20(3):323 334, 2007. ISSN 08936080. doi:
Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7 (1):13276, 2016. ISSN 2041-1723. doi: 10. 10..01////mmmm12220
Zhuo Liu, Hao Shu, Linmiao Wang, Xu Meng. Yousheng Wang. Xuancheng Li, Wei Wang, and Tao Chen. Adjoint propagation of error signal through modular recurrent neural networks for biologically plausible learning. eLife, 15te108237, 2026. ISSN 2050-084X. doi: 10.7554/eLife. 108237.
106237,
Adam S. Lowet, Qiao Zheng, Sara Matias, Jan Drugowitsch, and Naoshige Uchida. Distribu- tional reinforcement learning in the brain. Trends in Neurosciences, 43(12):980-99) 2020. ISSN 0166-2236. doi: https://doi org/ w0//10.100000000 10100..111...iii 2020.09,004 URL https: /wwww.
sciencedirect com/ www../ow....ce.eeeeeee.eeeeeeeee swwesciieecceeeeeeeeeeeee wtt.o///w.....eeerrr..e////////.
Christopher W. Lynn and Danielle S. Bassett. The physics of brain network structure, function
and control. Nature Reviews Physics, 1(5):318-322 2019. ISSN 2522-5820. doi: 10.1038/
019-00088
Nikola T. Markov, Mária Erasey-Ravasz, David C. Van Essen, Kenneth Knoblauch, Zoltán Toroczkai, and Henry Kennedy. Cortical high-density counterstream architectures. Science, 342 (6158), 2013. ISSN 0036-8075 1095-9203, doi: 12t///ciecc..123333333 1238406.
Jorge F. Mejias, John D. Murray, Henry Kennedy, and Xiao-Jing Wang. Feedforward and feedback frequency-dependee interactions in large-scale laminar network of the primate cortex. Science Advances, 1)11136633335, 2016. doi: 10. 10.i:10...1//ciiiii 126/sciaav. 1601335. URL https:: //tp:://www
science. .oanice.rre ortp:/e/rrr//////bbbbb
1601335
Jan Melchior and Laurenz Wiskott. Hebbian-descent, 2019. URL https://erxivw
19C5.
Alexander Meulemans, Matilde Tristany Farinha, Javier Garcia Ordonez, Pau Vilimelis Aceituno, João Sacramento, and Benjamin F. Grewe. Credit assignment in neural networks through deep feedback control. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4674-4687 Curran Associates, Inc., 2021. URL httttps:/pprocceeeieee meur www.:ne.rrcccccccooo..... cc /e 5wf (at.eoofofooccc-aPPaaaee
fi
Georgios Michalareas, Julien Vezoli, Stan van Pelt, Jan-Mathijs Schoffelen, Henry Kennedy, and Pascal Fries. Alpha-beta and gamma rhythms subserve feedback and feedforward influences among human visual cortical areas. Neuron, 99(2):344-397 2016. ISSN 0896-6273. doi:
Sut. URL //www//wwww///../. sciencedirect. com/
12

---

Published as a conference paper at ICLR 2026
John Miller and Moritz Hardt. Stable recuirent models. In International Conference on Learning Representations, 2019. URL
Beren Millidge, Yuhang Song. Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. Back- propagation at the infinitesimal inference limit of energy-based models: Unifying predictive cod- ing, equilibrium propagation, and contrastive hebbian learning. The Eleventh International Conference on Learning Reprexentations, 2023, pttptttss://ppnreeeet
net/ forum?
id=nIMi fq22E0
Javier R. Movellan. Contrastive Hebbian Learning in the Continuous Hopfield Model, pp. 10-17. Morgan Kaufmann, 1991. ISBN 978-1-4832-14497 doi: https://doi.org/10..uu wwtpwwscienceee http://wwww..eeeeeeeeeeeeoooeeea
B978- 1-4832- 1448- 1,50007-X. URL
article/p1119888881444433
Mitsumasa Nakajima, Yongbo Zhang, Katsuma Inoue, Yasuo Kuniyoshi, Toshikazu Hashimoto, and Kohei Nakajima. Reservoir direct feedback alignment: deep learning by physical dynamics. Communications Physics, 7(1):411, 2024. ISSN 2399-3650. doi: 10. (0.10088/442005-04--018993
Arild Nøkland. Direct feedback alignment provides learning in deep neural net- works. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https:://procee nww. #111/499001447772258800000111111 -at.:eaacciaaaa--aae.e.... pdf
ngs. nwwrinuurr...e wwwpsscccpppee wwtpccprppacpceeeeeeeee www.ff/eppppe///00000/ 6/
Peter O'Connor, Efstratios Gavves, and Max Welling. Initialized equilibrium propagation for backprop-free training. In International Conference Learning Representations, ICLR, 2019.
URL /tpp:ttt////epeeeeiewww net www.teetttoooee wtw.:/oddmmmmmmiiiimaa
Alexander Ororbia and Ankur Mali. Biologically motivated algorithms for propagating local target representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp- 4651-4658. AAAI, 2019.
Alexander Ororbia, Patrick Haffner, David Reitter, and C. Lee Giles. Learning to adapt by minimiz- ing discrepancy, 2017.
Alexander G. Ororbia. Brain-inspired machine intelligence: A survey of herorrrooonnuohhoooollylyyaaaaaa credit assignment, 2023. BRLhhtts:: /ttpss///rrases //trx/aaxivvvv//.a//.aa wtt.aaawwww..ga./aaa//aaeeeaeeee www./orgabs..................... http://bs///......///ee 09257
Alexander G. Ororbia, Ankur Mali, Daniel Kifer, and C. Lee Giles. Backpropagatioon-fre deep learning with recursive local representation alignment. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9327-9335. AAAL, 2023. URL attps://oj.aaaaii org/ org/index index. php/ Awtp:/AAI//aaaa wwt.A///attieee/eee 211.1///11/////111
Matthew G. Perich and Kanaka Rajan. Rethinking brain-wide interactions through multi-region 'network of networks' models. Current Opinion in Newrobiology, 65:146-151, 2020. ISSN
09594388. doi: 10.1016/j.con. 2020.. 11.003.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back propagating errors. Nature, 323(6088) 1986. ISSN 0028-0836. doi: 10. 10110/3353333.. URL
<GotoIsI>: : TE0.99886337000 Count:4.
E3275 Times Cited:1672) Cited References
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic corti- cal microcircuits approximate the backpropagation algorithm. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Bianchi Bianchi, and R. Garnet (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL. httpps://procca file/1  1ttee//1233000000000020202023223 atw.a:122200bbaaaaaaaaaaaeeeeee httpl/ppperfffespppppppeeepppppe
i ngs. .etrssnnuriiee nwwr www.ris...c.pppaeeeea.e. fi fil.filllleee less/pppe//21....
Tommaso Salvatori, Luca Pinchetti, Beren Millidge, Yuhang Song, Tianyi Bao, Rafal Bogacz, and Thomas Lukasiewicz. Learning on arbitrary graph topologies via predictive coding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Informa- tion Processing Systems, 2022. URL
net hwt.:nnnettooaaeeeeeee forum? wiw.ummdddddddnnn.
R9A
13

---

Published as a conference paper at ICLR 2026
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in Computational Neuroscience, 11:24, 2017. ISSN 1662-5188. doi: (0.3389/mcoo.2217.0002.
Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, and Yoshua Bengio. Gener- alization of equilibrium propagation to vector field dynamics, 2018. URL https://arxxy
org/ 1808. 04873
Benjamin Scellier, Maxence Eroullt, Jack Kendall, and Suhas Kumar. Energy-based learn- ing algorithms for analog computing: a comparative study. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 52705-52731. Curran Associates, Inc., 2023. URL httpps:/pproceednngss neurips. cwwppperrrccpppprr.rpeeeep. http://erreiieeeepeeee//// File/ 32114488000000337777000000004-00
pdf
João D. Semedo, Anna I. Jasper, Amin Zandvakili, Aravind Krishna, Amir Aschner, Christian K. Machens, Adam Kohn, and Byron M. Yu. Feedforward and feedback interactions between visual cortical areas use different population activity patterns. Nature Communications, 13(1):1099, 2022. ISSN 2041-1723. doi: 10.1038641447--222 285522.... URL httphttt/do....oo
Shawn C. Shadden, Francois Lekien, and Jerrold E. Marsden. Definition and properties of la- grangian coherent structures from finite-time Yyapunov exponents in two-dimensional aperi- odic flows. Physica D: Nonlinear Phenomena, 212(3-4):271-304 2005. ISSN 01672789. doi: 1.100166..hysd122051000000
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpt::s:///proeeednggs....e Taebee/2233aeee11rooooooooaoaaaa
Andreas Veit, Michael J Wilber, and Serge Belongie, Residual networks behave like ensem- bles of relatively shallow networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gareett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL httpps///prroeeddiggg..ee....o. Gotp://DAAaaaP7......iceeeaa2212
files/
Ran Wang, Xupeng Chen, Amirhossein Khalilian-Gourtani Leyao Yu, Patricia Dugan, Daniel Fried- man, Werner Doyle, Orrin Devinsky, Yao Wang, and Adeen Flinker. Distributed feedforward and feedback cortical processing supports human speech production. Proceedings of the Na- tional Acauemy of Sciences, 120(42):e200255512, 2023, doi: 10. 1...073/paas23302255 230055512... URL hitpps://do www.orgg...... 1073 /073/pnas wttppmas...0005551 2300255120
doi: 10.
D. J. Watts and S. H. Strogatz. Collective dynamics of 'solll-worldd networks. Nature, 393 (6684):440-442, 1998. ISSN 0028-0836. doi: Doi 1038/30918. URL <GotoIII:://RSSS
00C074020000035 Zr842 Times Cited:29354 Cited References Count:27.
Alan Wolf, Jack B. Swift, Harry L. Swinncy, and John A. Vastano. Determining іyaрunow exponents from a time series. Physica D: Nonlinear Phenomena, 16(3):285-317, 1985.
URL http::tttt////t sciencedirech /tt.e htt.://tww.......eoaaaa.aaaaaaaa
ISSN 0167-2789. doi: http://da.oog/000.00000077777910
X. Xie and H. S. Seung. Equivalence of backpropagation and contrastive hebbian learning a
layered network. Neural Computation, 15(2):441-444 2003. ISSN 0899-7667. doi: 10.1162/ 0899766037625522988
Izzet B. Yildiz, Herbert Jaeger. and Stefan J. Kiebel. Re-visiting the echo state property. Neural Networks, 35:1-9, 2012. ISSN 08936080. doi: 10. 2..00166.eeunet.2112.077
14

---

Published as a conference paper at ICLR 2026
A THE DYNAMICS OF THE RNN
We quantify the convergence property of the recurrent neural network (RNN) with maximum Lya- punov exponent (MLE) (Wolf et al. 1985), and finite time maximum Lyapunov exponent (FTMLE) (Kann & Uchida 2014). When the MLE/FTMLE is large, the RNN converges slow or even not all. To compute MLE and FTMLE, we first initialize random perturbation vector 6g. Then record the sequence of states s with / = O, 2,2,... ... Te - corresponding to the last sample of training set (see Figure 2 in the main text), and the following steps:
1. Normalize perturbation vectors to unit length:
52
110
2. Calculate the Jacobian matrix:
[(8 = OF (F
[
3. Update the perturbation:
5.41 = [s) -6
4. Record
ri In 11xc+11
The maximum Lyapunov exponent is computed Amax 21=0 T; for a sufficiently large Te (default T. 500). The results any T < T. the FTMLE.
Figure ST S2 show the FTMLE, MLE, training accuracy and test accuracy versus epochs of dif- ferent models. In all cases, smaller 3; usually yields smaller (FT)MLE, whereas larger do always lead to larger (FT)MLE because the activation function saturates. The saturation diminishes perturbation.
For 2-hidden-layer RNN, smaller feedback scaling 3g yields steady training progress and better ac- curacy. (S the FTMLE and test accuracy against feedback scaling for different numbers of hidden layers. It shows that smaller 3; is favorable for shallow networks, because the RNN is eas- ier to converge (indicated by FTMLE). But for deeper networks (5-hidden-layee or more), smaller 33 degrades performance because of vanishing gradient.
Further comparison between our FRE-RAN RNN that incorporates convolutional strucutre with previous work (Ernoult et al. 2019) are also plotted in Figure (S4] These results suggest that small feedback scaling (3: 0.01) leads to a smoother training process.
B GRADIENT VANISHING AND THE RESIDUAL CONNECTIONS
Figure $5 and S6 plot the error of each neuron versus epoch different 3, For 2-hidden-layer RNN, the best performance is obtained at 3; = 0.001. In this situation, the error of the first hidden layer is at least two orders of magnitude less than the second hidden layer. At 3, 2, the error also decreases from higher (high index neurons, closer to output layer) to lower layers, which is attributed the saturation of the activation function. general, the training progresses more steadily for smaller 8; despite the vanishing gradient, which also applies to deeper networks (up to 10-hidden- layer).
To eliminate the vanishing gradient in EP, direct feedback from the higher layers or local amplifi- cation (with higher learning rate) is unavoidable (Nokland) 2016,) Ororbia et al.) (003). Figure SS7 shows the effect of residual connections. 8, = 0.1 yield the best accuracy 97.5%, due to the balance between gradient flow and convergence.
Figure S8 shows the testing accuracy varies with the connection probability P of AGT with 10 hidden layers. Except for the connections in layered model, the connection between any two hidden layers is generated with probability P, i.e., we first use P to decide if the connections between any two layers will be established. As P increases, the accuracy rises first, peaks at 0.2 and decreases around However, the reason behind is yet to be explored.
15

---

Published as a conference paper at ICLR 2026
ancuray
0.8
0.8
MIE
0.4
20
20
20
20
Epach
Epoth
Epech
Epach
3.0
1.0
accuracy
0.001
ITMLE
0.01
0.1
0,25
0.0
0.0
20
20
Epoch
Spech
Spech
Spoch
M
ITHLE
0.01
20
20
20
20 Epech
Epoch
Epech
Epech
3.0
sccurasy
0.0
0.0
MILE
0.4
0.25
0.0
0.0
Epoch
Epech
Figure S1: The FTMLE, MLE, training accuracy and testing accuracy of symmetric RNNs versus epochs with different feedback scaling 3 (legend). First 2 hidden layers; Second row: 3 hidden layers; Third row: hidden layers; Fourth row: 10 hidden layers. The activation is tanh. Each case repeated 5 times.
16

---

Published as a conference paper at ICLR 2026
0.0
0.8
atcurace
0.0
0.4
0.01
4.0
0.25
0.0
0.0
accuracy
0.8
accupary
0.8
0.6
0.4
20
20
20
20
Epach
Epoth
Spoch
tpoch
1.0
1.0
acturscy
10.6
0.6
0.01
0.001
2.0
1.0
0.25
0.2
0.2
20
40
20
40
20
40
20
40
Epoch
Epoch
Epach
Epech
1.0
1.0
accurnay
10.6
0.6
0.001
D.01
0.4
0.2
0225
0.0
20
20
20 Spach
20
Bpach
Spach
Epoch
Figure S2: The FTMLE, MLE, training accuracy and testing accuracy of asymmetric RNNs versus epochs with different feedback scaling (legend). First row: 2 hidden layers; Second row: 3 hidden layers; Third row: hidden layers; Fourth row: 10 hidden layers. The activation is tanh. Each case repeated 5 times.
(b)
0.0
0.8
0.8
00.5
Testing scourscy
FINE
-.00
0.6
-15
-2.0
-25
0.2
+30
10
0.2
0.25
B.
Figure S3: The FTMLE and testing accuracy versus feedback scaling 3 with different numbers of hidden layers. (a) Symmetry weights; (b) Asymmetric weights. The FTMLE and testing accuracy given here correspond to their maxima in all epochs. Note that the 5-hidden-layer asymmetric RNN with large & diverged and resulted in missing data points in (b). Each case is repeated 5 times.
17

---

Published as a conference paper at ICLR 2026
(b)
8.0000
1.50
0.98
0.88
0.54
0.92
0.92
10
25
30
33
10
10
11
Epoc
(c)
8,10,1
(d)
6-11
3.50
1.00
traviigg
0.75
0.75
0.30
0.50
0.25
0.25
Figure S4: Comparison of RNN embedded with convolutional structure the MNIST between P-EP (a) (Emoout et al., 2019) and our approach at different B3 (b-d). used the same parameters as the EP reference (Ernoult et al. 2019).
18

---

Published as a conference paper at ICLR 2026
Error L2 0
Accuracy L2 8,00001
100
1.00
(b)
Neuron index
10-2
0.95
Accuracy
50
10
0.90
100
10
0.85
traning
test
0.80
0
10
20
30
40
10
20
30
40
50
Epoch
Epoch
Error_ Error_L2__ 0.0000
Accuracy 0.1
100
1.00
(c)
(d)
Neuron index
10
Accuracy
50
10
0.90
100
10-6
0.85
traning
10
0.80
20
Epoch
Epoch
Error Error_L2_A 12_00=110000
Accuracy 12_8,=1.0
10c
1.00
(e)
(f)
traning
Neuron index
10-2
0.95
Accuracy
10
0.90
100
10-6
0.85
10-9
0.80
0
10
20
30
40
10
20
30
40
50
Epoch
Epoch
Error L2 8,=2.0-0
Accuracy L2 9,=2.0
100
1.0
(h)
traning
test
Neuron index
10-2
0.8
Accuracy
50
0.6
10-4
0.4
10-6
0.2
10-8
0.0
0
10
20
10
Epoch
Epoch
Figure S5: For 2-hidden-layer RNN, the mean error of each neuron in the last batch and testing accuracy versus epochs at different 3,. All neurons in the hidden layers and the output layer are indexed from the input to the output layer.
19

---

Published as a conference paper at ICLR 2026
Error Error_L1o_fae L10_8,=0.001-0
Beturacy_L1o_A-00000
10
1.0
(a)
(b)
Neuron index
10-2
0.8
Accuracy
0.6
10
400
0.4
10
0.2
600
traning
test
0.0
0
10
20
30
40
10
20
30
40
50
Epoch
Epoch
Error L10_ 00110
Accuracy_LLD 10011
100
1.00
(c)
(d)
traning
test
Neuron index
10
0.95
200
Accuracy
10-4
0.90
400
10-6
0.85
600
10-8
0.80
20
Epoch
Epoch
Error L10 1110,,1=1.000
Accuracy L___y_11111000000
10c
1.0
(e)
(f)
traning
Neuron index
10-2
0.8
200
Accuracy
0.6
10
0.4
10-6
0.2
600
10-8
0.0
0
10
20
30
40
10
20
30
40
50
Epoch
Epoch
Error Error_L10_8,-10 2
Accuracy ______10__22211____
10
1.0
(h)
traning
test
Neuron index
10-2
0.8
200
Accuracy
0.6
10-4
400
0.4
10-5
0.2
600
10
0.0
0
10
20
40
10
20
30
40
50
Epoch
Epoch
Figure S6: For the 10-hidden-layer model, the mean error of each neuron in the last batch and testing accuracy versus epochs at different 3,, All neurons the hidden layers and the output layer are indexed from the input the output layer.
20

---

Published as a conference paper at ICLR 2026
Error Error_L10_B,=aa _10108,000000-000
Accuracy_LDD Btapyy___11____11 8,0000
10
1.00
(b)
traning
Neuron index
10
0.95
200
Accuracy
10-4
0.90
0.85
600
10-8
0.80
10
20
Epoch
Epoch
Error L10 111_8,-0.14
Accuracy L10 0.1
0
10c
1.00
(c)
(d)
Neuron index
10-2
0.95
Accuracy
10-c
0.90
400
10
0.85
600
traning
test
10-9
0.80
0
10
20
30
40
10
20
30
40
50
Epoch
Epoch
Error L10 8,=1.0-0
Accuracy L10 8=1100
10
1.00
(f)
Neuron index
10-2
0.95
200
Accuracy
10-4
0.90
10-6
0.85
600
traning
test
10-8
0.80
Epoch
Epoch
Figure S7: For the 10-hidden-layer model with residual connections, the mean error of each neuron the last batch and testing accuracy versus epochs at different 8, All neurons in the hidden layers and the output layer are indexed from the input to the output layer.
0.95
Testing Accuracy
0.90
0.80
0.0
0.02
0.1
0.2
0.5
1.0
Figure S8: The testing accuracy on MNIST varies with the connection probability of AGT with 10 hidden layers. The experiments repeated 5 times.
21

---

Published as a conference paper at ICLR 2026
Table S1: Testing accuracy (mean of 5 repeated experiments) with different feedback scaling 3 By
default, T = 10 x LaddenLayer K = 5 x Haddee Each hidden layer has 64 nodes.
Architectur--comectionn B, 0.001
3, 0.01
8 = 0.1
B, 0.25
B =
B, =
8,
2HLmsymm
97.69%
97.57%
97.25%
96.22%
93.12%
66.04%
40.92%
3HL-symm
97.22%
97.64%
97.41%
96.60%
55.86%
32.64%
22.11%
5HL-symm
93.54%
95.54%
97.60%
90.63%
25.31%
17.88%
14.61%
10HL-symm
87.15%
89.99%
92.54%
41.84%
14.07%
14.30%
14.23%
HeIL-Residual-yym
97.52%
97.46%
95.51%
conv-symm
99. 19.15%
98.71%
11.35%
2HL-asymm
96.96%
96.97%
96.88%
96.79%
93.88%
91.81%
89.91%
3Hymasymm
95.17%
96.91%
96.76%
96.66%
91.21%
54.65%
26.72%
SHL-asymm
91.14%
92.34%
96.41%
96.35%
17.15%
11.35%
13.07%
10HL-asymm
84.27%
85.83%
87.79%
90.97%
16.13%
14.21%
16.67%
10HL-AGT-asymm
96.37%
96.75%
33.31%
C EQUIVALENCE WITH EP AND BP UNDER THE CONDITION OF INFINITESIMAL INFERENCE LIMIT
So
S1
S2
Predict: Sp
Label:s
Wo
W1
Wi
00O
2OOD
11 B1
By B
Figure S9: A layered network model used to illustrate the process of backpropagation (BP), local representation alignment (LRA), and EP. Note that the final prediction layer "p corresponds the third layer with subindex "3- For LRA, we use BLRA instead of 31 and By. For BP, the feedback (orange) paths are absent.
In this section, we will use the infinitesimal inference limit (Milldgge et al. 2023) to derive the cquivalence of EP with LRA and BP.
C.1 BACKPROPAGATION
When we remove the feedback connection of 2-hidden-layer RNN shown in Figure S9 feedfor-
ward network is left and can be trained with BP. The forward process of BP described by:
p(hi), hy = Wo 80:
82 p(h22. ho W1 511
(SI)
Sp hps hp = W3
Defining a loss LBP AW, as an example:
(0p Star)² 2, the weights adjust according to the gradient of the loss. Taking
AWr OLBr
aW.
= O WTT (p'(122 o W.T. (sp Stor)) (so)
(S2)
where "O" means Hadamard product (element-wise product), "." means scalar or matrix multipli- cation. For two vectors/matrices, "ø" requires identical dimensions and computes element wise products. Broadcasting rules may apply (c.g.. column vector unx1 @ Amxn scales each column of A by v).
22

---

Published as a conference paper at ICLR 2026
C.2 LOCAL REPRESENTATION ALIGNMENT
LRA is an alternative training method following the principle of discrepancy reduction (Ororbia al. 2017 Ororbia & Mali 2019). can be divided into phases: 1) the network the forward process, producing latent representations of the input samples. 2) the weights adjust in the direction of reducing the mismatch between current latent representations and target representations in each layer.
The forward process is the same as BP:
d = p(h)),
hy Wo S0,
82 p(h2),
h2 W1 8%.
(S3)
sp pp
he WF s9.
where sf are interpreted as the latent representations. The prediction error is Cp Star we can get the target representations of the second hidden layer:
Sp. Then
ALRA = p(h 1/2 f = W1 of + BLRA By Ep.
(S4)
The same goes for the first hidden layer:
$1 QQLnA
p(h (
h I W1 3o + SLRA B1 -2.
62 = 52
BlnA
$2
(S5)
LRA defines the loss as the total discrepancy between latent representations and target representa- tions:
LLRA Σ L,L,
SURRA)
Σ 70 SuRAA
(S6)
The weight W; adjusts according to the local mismatch between 24t and BURA. 8;41
ak, Li,L1 PLRA 81+1
AW, =
OW;
BLAA 2441) O (( (6
(5
R3 (sftn
(S7)
where the derivative of the activation function is omitted in the last row, useful practice common LRA (Melchior & Wiskott 2019 Ororbia Mali 2019 Ororbia et al. 2023). When SLRA + 0,
BLRA
and h MLaA - h. then
BLNA 8 = p(h p(h))
6 = 8
= p(h + +LRA - B, (4411 - p(h())
= [p(h)) + +p((1)) @ (BERA Bd ei+1) - -((h))
(S8)
-'(h)) @ (BLRA -x+1)
The approximation in Equation S8 is based on first-order Taylor expansion of p(h + Ah) around hg. where Ah- = BLRA B. + For a small perturbation ah - the Taylor expansion gives:
p(h@ Ah) p(h)) + p' (((")) Ah + O(A)?),
(S9)
When BLRA O. higher order terms O(Ah²) are negligible, leaving only the linear terms. We arrive at the last row after canceling out p(hg). There we can express the weight adjustments as
A W. = (+0))
= (p'(h() & BLRA B (p'(12) o (BLRA By (Stor sp)))
(8o) (::
-BLRA PLRA ('( @ W.T. (p' @ W.T (sp star) (so)T,
(S10)
which is the same as BP (Equation S2 except for a constant. Thus, LRA at weak feedback limit approximates BP. An LRA algorithm for 2-hidden-llyee network is described in Algorithm ST The feedback weights in LRA need not be learned here, but can be kept symmetric with the feedforward weights.
23

---

h.: Published as a conference paper at ICLR 2026
C.3 EQUILIBRIUM PROPAGATION
We can also formulate EP in terms of discrepancy reduction. In EP (Algorithm in the main text),
the network states evolve as follows (3 = for the first phase and B = By for the second phase):
h Wo- 5 31 B1 $2
h2 Wi o + 35. By Epp
k W .82
3
8 3 = 1 2 4p
(S11)
where ep Star sb is the predicting error. The network converges final states h., h2, s1. sg in the free phase. The error of 32 neurons can be described by:
ds2 = [p(h) 118 - [(((h))) ~ (1())) @ (3y By 6p),
(S12)
where only the first-order infinitesimal term is retained as 31 - 0. The same goes for the first hidden layer:
dsi =
(1hh))
= p'(i)) o (3: B₁ (0'(1g) o (3, (p.c)))))
(S13)
The weight Wo can be updated by:
dsi (sf)
AW
(b @ Bi (p'((h)) @ B,.,,) (s8)
(S14)
B1 3g
With B, W.T.
dsj = By 31 p((h
@ WWT (d'(h)) o WT -(sp - ssarr)),
(S15)
dsp (58)
AW
B1 B5 -p'(1))) wiT (p((h)) o WT (sp Star)) (T8)
(S16)
Note that compared with the weight update in the main text, 1 added to recover gradient amplitude similar to BP. Further, if we assume that the high-order infinitesimal in the first phase can be omitted, the dynamics of RNN is governed by:
" p())))
hy W. + 31 B
82 +0 R Wo 80,
0
(S17)
52 = p(hk),
h
W1 s.
81 B -p W1 sf.
(S18)
ho p h W 82-
(S19)
p
The information flow of RNN degenerates into that of a feedforward network. This does not affect the error information ds, thus Equation S16 approximates Equation S2 for BP. Meanwhile, it re- sembles LRA with low SLRA. which turns explicit error into implicit error. Hitherto, have shown that although the errors are obtained differently in EP, LRA, and BP, they are equivalent under the assumption of weak supervision and weak feedback.
24

---

Published as a conference paper at ICLR 2026
Algorithm S1 Local Representation Alignment (LRA)
Input: (I, Star)
Parameter: 0 = [Wo. W1, W2, B2, B1, SLRA]
Output:
1: function FORWARD(O0
2:
50
3:
- p(h1),
h1 + WD
4:
- p(h2).
h22 W1
5:
- Wy.
6: A < [sf], i 0, p,
7: return A1
8: end function
9: function FEEDBACK(O, A1, Star)
10:
Cpt Star - sp
11:
$2 BLRA t -((2)))
112 + 51 BLRA Bg.0. op
BunA - $2
12:
e2 t 82
13:
$1 SERA -p(h)),
h1 Wo 80 BLRA. B2
61 + s ALRA 59
15:
AS + e2, Ep.
16: return A2
17: end function
function UPPATING-WEIGIITSS A1,A2)
19:
AW; & 6i+1 (
i= 0,1
20: AW, 4- Ep (ss)
21:
end function
C.4 EXPERIMENTS FOR EQUIVALENCE WITH EP AND BP
Prior works have shown that EP can be equalized to BPTT in specific conditions and can achieve comparable performance (Ernoult al. 2019: Laborieux al. 2021). As discussed in the previ- ous section, although the overall architecture forms an RNN, the network behaves similarly to feedforward model due weak feedback connections.
To experimentally show the equivalence of EP and BP, we can further compare our model with FNN with same feedforward weights trained by BP. We mainly compare cosine similarity of states, bias gradients and weight gradients for the first batch (batch size is 200) as given in Figure S10 Tigure/S100a-1 a-c) shows similarity under the conditions of B3 with different iterations. For the the bias gradients, i.e., ds; the cosine similarity declines rapidly. indicating no similarity between our model and BP. With weak feedback B, 0.1, shown Figure d-f), the similarity of states approaches 1 and the similarity of bias gradient of last 6(4) layers exceeds 0.5 with T = 500/50 (T=200) 20). These results provide further evidence that EP is equivalent to BP under the condition of weak feedback.
We further studied the influence of 3, on the cosine similarity. Figure S10 g) shows that lager 3: leads to lower similarity of states. Figure hS10h) shows that lower 3; 0.01 also leads to the decrease in similarity, which may caused by insufficient precision of data storage (float32 by default). Therefore,
use datatype float64 to repeat experiments. Figure SIO shows that the similarity of gradient signal remains around with 8 = 0.1. This indicates that weak feedback does indeed lead to exponential decline in gradient signals, thus requiring higher relative accuracy.
25

---

Published as a conference paper at ICLR 2026
(a)
states
(b)
bias gradients
(c)
weight gradients
1.0
1.0
1.0
B,-500
sim aaity
cosine similartty
cosine similaityy
8==50
0.5
8,=20
0.0
(2
12
(c3
AcAAA
129
166
t27
1c8
out
(22 42> ych
(2
(2)
tcB
out
(c3
(2
123
1c
166
teB
out
(d)
states
(e)
bias gradients
(f)
weight gradients
similarity
cosine similarity
cosine similarity
0.5
0.0
8=50
2-
-0.5
-0.5
-0.5
1c²
102
1c3
fcA
105
jcb
Fc
1cB
out
(ch
102
12
sch
166
1cB
out
1C1
10
103
sc
106
(c)
tcB out
(g)
(h)
blas gradients
weight gradients
1.0
1.0
1.0
B A
B 0.5
similarity
cosine similarity
cosine simiariky
6 0.01
0.5
0.5
0.5
B=:0.1
0.0
0.0
0.0
-0.5
-0.5
-0.5
122
103
105
1c0
out
(23
166
1c8
out
[c)
122
(23
166
xc
(cB
out
(i)
states
(k)
bias gradients
(1)
weight gradients
1.0
1.0
cosine similaitty
cosine simiarity
0.5
0.5
simi
0. 0.
=
0.0
0.5
A==1
(0
162
103
166
108
out
10
122
123
106
1cB
out
(0)
12
123
1ch
166
(c)
teB
out
Figure S10: The cosine similarity of gradients and states between our model and feedforward model trained by BP in an hidden-layer FNN (states: sQ: bias gradients: ds; weight gradients: AW)). The axis is the layers of the model. Error propagates from the last layer "out" to the first hidden layer "fc1" layer by layer. (a-c), with different numbers of iterations under feedback scaling 3, = 1. (d-f), with different numbers of iterations under small feedback scaling 8 0.1. (g-i), with different feedback scaling (T=50,K=20). (j-l), Repeat (g-i) with datatype float64 (float32 by default).
26

---

Published as a conference paper at ICLR 2026
C.5
VERIFYING THE EFFECTIVENESS OF WEAK FEEDBACK IN EQUILIBRIUM PROPAGATION
1.00
8,=0.01
0,1
0.95
Test accuracy
0.90
0.85
0.80
Only last ny layers learning
Figure S11: The testing accuracy on MNIST with different 8 varies with ny. The experiments are repeated 5 times.
To demonstrate that the lower few layers of our model are indeed receiving meaningful credit signals, report the test accuracy of only updating the last nu layer (i.e., freezing the weights of Layers 1 1- nu) in Fig For 5-hidden layer model with 3, 0.1, updating only the final layer yields test accuracy of about 85%. As ny increases to the accuracy also reaches around 97.5%. A similar trend is observed for the model with 3 0.01. These results show that achieving over 97% accuracy requires effective gradient propagation to all layers, confirming that our model successfully delivers usable credit signals throughout the entire network.
C.6
ROBUSTNESS TO THE NOISE
(a)
(b)
(c)
1.00
1.00
1.00
Weigt Interdiy
0.55
0.55
0.95
0.001
0.972
0.972
8.972
0.894
0.001
0.923
0.935
0.060
0.749
0.001
0.473
0.812
0.31
0.917
0.917
0.912
0.689
0.85
0.01
0.903
0.504
0.838
0.485
3.85
6.01
0.876
0.815
0.520
0.299
0.85
0.80
0.80
0.80
0.194
0.196
0.217
0.150
0.367
0.134
0.75
0.75
0.75
1e-05
0.001
0.1
1e-05
0.001
1e-05
0.001
State nokse Inteesity
State nolve Internity
molse Intensiity
Figure S12: The maximum test accuracy model with different noise intensity weights and states added both training and test. (a) With 2 hidden layers; (b) With hidden layers; (c) With 5 hidden layers. The model is trained for 50 epochs and the experiments is repeated 5 times.
To evaluate the robustness of the model, we introduce noise on weights and time-varying noise on states, which are random Gauss noise imposed at each weight update or at each state update, respectively. The noise on weights directly added the weight, while the noise on states is added as the bias b in Equation 3.1 The mean absolute values of non-zero weights and neural activations after noiseless training are approximately 0.09 and 0.76 respectively.
The accuracy of the model with two hidden layers varies with two types of noises as presented in Fig- ure a)) maintains satisfactory performance when the standard deviation state noise reaches 0.1 or the standard deviation of weight noise reaches 0.01. In deeper structures (Figure (I)), the results are consistent with the aforementioned observations for weight noise, demonstrating ex- cellent robustness. However, the tolerance to time- varying state noise degrades significantly, which we attribute to the laser-wise noise accumulation and the distortion of weak gradient signal by the noise in the training process. To confirm hypothesis, impose the noises only in the test,
27

---

Published as a conference paper at ICLR 2026
(a)
2HL
(b)
3HL
(c)
SHL
1.00
0.973 0.971
0.971
3.811
0.97
0.976
0.975
0.000
0.975
D.977
0.97Y
0/977
0.918
wwigh 1ubb entreek
0,001
0:971
0.971
0,974
3.970
3.945
0.90
0.001
0.975
0:973
0.973
0.972
0.945
0.90
0.002
0,373
0.973
0.973
0,972
0.928
0.90
0.919
0.936
0.939 0.518 0.512
0.85
0.01
0.903
0,908 0.8 0.902
0.892
2.85
0.876
0.880
0.873
0.842
0.85
0.80
0.80
0.00
0.3 0.99 0.157 0,134 3.219 3.177
0.1
0.175
3:182
0.152 0.157 0.142
0.3
0.1-0:133
0:128 0:143 0.153 0.142
0.001 0. 0.001 0.1
0,001 0,001 0.3
0.5
0 0.001 0.001 0.1
0.5
State noise intenityy
2,00 RAGS MEnrAry
Figure S13: The maximum test accuracy model with different noise intensity on weights and states (the state noise is added only in test). (a) With 2 hidden layers; (b) With 3 hidden layers; (c) With 5 hidden layers. The model is trained for 50 epochs in a single experiment.
and the test accuracy almost remains unaffected (Figure S13 Therefore, the network is potentially resilient to noise. However, how to improve resilience in the training process require further study.
D TRAINING DETAILS
Table S2 provides the parameters of the Adam optimizer that are used in Tables S2 S2
Diederik
Kingma 2015). The training details for Table given in Table S3 For convolutional architec- tures in EP,, the training process can be described by Algorithm S2 The training sample is fed into the network through Convo. Then the state of the first layer goes through max pooling MaxPool, and convolution Conv, sequentially to reach the second layer. The second layer also feedbacks its states to the first layer through transposed convolution ConvT, and max-unpooling Max Unpool.. With T iterations, the RNN converges to the steady states and produces outputs through MaxPool, and a fully connected layer. Then the prediction error is computed and used to nudge the RNN by the reverse of the fully connected layer and max-unpooling MaxUnpool2- Note that the unpooling Maxoool,, requires the indices from the corresponding pooling MaxPool,.
For Table $2 Adam optimizer is used for all experiments. The activation functions sigmoid-s p(x) max(min(e 0), 1), respectively (Ernoult et al., 2019). For 5-HL, 10-HI. and 20-HL architectures, the Adam optimizer parameters are as shown in Table S2 (epoch: 50, batch size: 500). The inference details of the architecture shown in Figure 3b are described by Algorithm S3 The details for convolutional architectures are
and hard-sigmoid are defined as p(z) = T4c-4ta 4.a
given in Table SS3 and Figure (155515
S15) The cosine-annealing scheduler is used convolutional architectures for CIFAR-10 (Tmax=
50. Mminn = 100
For MNIST, no pre-processing is used. For the CIFAR-10 dataset, we follow ref. (Scellier et al.) 2023) to pre-process the images. We normalize the input images using mean (0.4914) 4, 4822, 0.4465) and standard deviation a x (0.2023, 0.1944, 0.2010).
The results for comparison of time consumption were obtained in a virtualized Windows 11 envi- ronment with Intel Xeon Gold 6238R CPU, 16 GB RAM,, and Nvidia RTX A,500 (24 GB VRAM). Other results were obtained on Windows 11 environment with Intel Core i5-12490F, 32 GB RAM, and Nvidia GTX 1650 (4 GB VRAM) or a Windows 11 environment with AMD R7-7700, 32 GB RAM, and Nvidia RTX 4070 (12 GB VRAM). The default numerical precision is float32 (single- precision float).
Table S2:
The parameters of the Adam optimizer. Parameter Name
Default Value
Learning rate (MNIST/ CIFAR-I0)
10 10-3/2 x 10
First-order moment estimation decay rate (81)
0.9
Second-order moment estimation decay rate (82)
0.999
Small constant for numerical stability (e)
10-
28

---

Published as a conference paper at ICLR 2026
Table S3: Training details for Table and Table The results of EB-EP and P-EP come from
previous work (Emoull et al. 2019). SGD refers to Stochastic Gradient Descent with mini-batcher
Epoch Batch size
Architecture
Training approach
Optimizer
Learning rate
Weight decay
-T/K
P-EP (sigmoid-ss)
SGD
50/20-100/20
(0,006, 0.05, 0.2)
None
2HL
Proposed (tanh, Adam)
Adam
50/500-10/10
[0.001, 0.001, 0.0001
None
P-EP (sigmoiddss)
SGD
100/20-180/20 180/20
0.002, 0.01, 0.05, 0.2
None
Proposed-DLa DLR (tanh)
SGD
100/20-18/10
00.002, 0.01, 0.05, 0.2
None
3HL
Proposed (uunh)
SGD
100/20-18/10
None
Proposed (tanh, Adam)
Adam
50/500-18/10
10-3
None
BP (tanh, Adam)
50/500-1/1
10-3
None
P-EP (hard-sigmoid)
SGD
40/20-200/10
0.015, 0.035, 0.15
None
Conv
(Table
Proposed (hard-sigmodd)
SGD
40/128-20/10
[0.15, 0.35, 0.9]
BP (hard-sigmoid)
SGD
40/128-1/1
10.001. 0.02, 0.4)
10
Conv
Proposed (hard thard-sigmoidd
Adam
40/128-20010
2 10
10
(Table
MNIST)
BP (hard sigmoid)
Adam
40/128-11 1/1
2 x 10-4
10
Cony
Proposed (hard sigmoid)
Adam
50/128-40V10
2.5 x 10
2 x 10
(Table
CIFRR-D 10)
BP (hard sigmoid)
Adam 50/128-1/1
2.5 10
2 10
646-xx8 8x8
64@4x4
32@24x24
32@12x12
1@28x28
1x10
Convi
MaxPooll
Con2
MaxPoo12
Dense
Figure S14: Convolutional architectures for MNIST.
128
12809404
64g/16x16
32032422
Figure S15: Convolutional architectures for CIFAR-10.
29

---

Published as a conference paper at ICLR 2026
Algorithm S2 Two phases in EP training process for convolution architecture
Input: Sample-label pairs (z, Stor)
Parameter: 0 = [Wo. W1, W a1, 81,
Output:
1: function FIRST-PHASE(O, Star)
2: 50
3: for t + 1 to do
4:
h1 Convo(se) + 31 MaxUnpool, (ConvT1(s9)
5:
h2 + Convi (MaxPool1 (sf))
6:
hp + W
7:
s). 22 sp + p(h1), p(h2), MoftMax(hp)
8:
end for
9: < [s]], i 1,2,p
10: return A1
11: end function
12: function
A1, Star)
13:
8/1 for
80. 81 -82 Sp
I. 81, s1
1 K do
Ep f- Star 5p 8n
15:
h1 + Convo(so) + 31 MaxUnpool, (Con)T((s?"11)
16:
17:
h2 +- Convi (MaxPool () + By-
Ciciaaooooouultatttnnnnnnn ep))
18:
hp W Flatenn(MaxPool((2"))
n Bn sp + p(h)), ((22), SoftMax(h))
19:
20: end for
21: end function
30

---

Published as a conference paper at ICLR 2026
Algorithm S3 EP with feedback scaling and residual connections (Figure 3p)
Input: (I, Star)
Parameter: 0 = [Wo. Wi, W,, B, Bi, Byi]
Output:
1: function ITERATION((, A1, Star)
2:
for & K do
3:
Nudging Phase then
4:
8 Bn 8r 8n
811
5:
80, 81
Sp +- I., 29,89.82,55 .82
6:
else
7:
31 +0
8:
Su 4- σ
9:
end if
10:
h1 - Wosy
+ B1s + B1,1 Bu.1s
h2 Wis 81 + +2
85
h3 - W280 81 +338384
12:
h4 W38 + +14BB4s
81 Wi, 31 + B7
14:
h3 + S + +33B588
h6 W
3, + 36
16:
h7 + W6s 37B858
+ W1,75'' 21 + 1100,B10,753
hg Wys, 31 Bgsss
hg + W8s 31 Bys
18:
h10 W989 + ByB W7.10s7
20:
hp + W1810
31 p(hi), 0=
21:
8,
3, Max(h))
22:
Sp
23:
end for
24: end function
31


Reviews:
[{"review_id": "dEu7DphQog", "reviewer": "Reviewer_Vmi3", "summary": "This paper introduces FRE-RNN, a biologically inspired recurrent neural network designed to make Equilibrium Propagation (EP) practical for modern deep learning. EP is a local, brain-inspired learning rule but has traditionally suffered from slow convergence and instability in deeper architectures. To address this, the authors introduce feedback regulation, scaling the feedback weights to less than 1 to stabilize dynamics and accelerate convergence. They further incorporate residual and random skip connections to improve gradient flow and enable deeper recurrent structures. Experiments on MNIST demonstrate that FRE-RNN achieves significantly faster runtimes than standard EP while maintaining accuracy comparable to backpropagation, and that it can successfully train deeper RNNs than previous equilibrium-based methods.", "questions": "1.\tIf the proposed model, which aims to closely mimic biological mechanisms, performs poorly on more complex datasets, what practical advantage does it offer over less biologically inspired models that achieve much higher accuracy?\n2.\tFor the results shown in Figure 5, were the reported trends averaged over multiple runs, or are they based on a single experiment? Can the authors conclusively state that lower beta values and higher alpha values consistently lead to better performance across architectures and datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper presents an interesting and well-motivated step toward developing biologically inspired yet computationally practical learning systems. The proposed FRE-RNN introduces effective modifications, such as feedback regulation and residual connections, that significantly improve the efficiency and stability of Equilibrium Propagation models. The authors demonstrate that their approach achieves at least an order-of-magnitude speedup in convergence while maintaining accuracy comparable to backpropagation-based networks. The experiments are clear, well-structured, and provide convincing evidence that these improvements make equilibrium-based RNNs more scalable to deeper architectures. Overall, the paper is clearly written, conceptually sound, and provides strong justification for the proposed modifications.", "weaknesses": "While the proposed framework demonstrates clear improvements over prior equilibrium propagation methods, it seems to only work well on relatively simple datasets. The approach performs well on MNIST but struggles on CIFAR-10, suggesting that it may not yet generalize effectively to more complex, real-world scenarios. Although the paper emphasizes the potential of bridging neuroscience and machine learning, it would benefit from a deeper discussion on the practical implications and real-world applicability of such biologically inspired models, especially given their current performance limitations. In addition, the experiments focus on image data, which is may not be the best fit for RNN-based architectures. Demonstrating results on temporal or sequential modalities, such as text, would have strengthened the case for the proposed approach. Finally, while the paper is generally well written, certain sections, particularly Section 4.1, could be improved for clarity and flow, as some results are presented in a non-linear order.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Yw0vP6AbD", "reviewer": "Reviewer_TKZt", "summary": "Equilibrium propagation is a biologically plausible method for credit assignment in deep neural networks. It operates through two phases, each requiring convergence to a fixed point. However, this relaxation process becomes exponentially slower as network depth increases. This paper addresses this problem by weakening feedback connections and introducing skip connections. The authors then demonstrate empirically that these modifications reduce convergence time and thus improve performance.", "questions": "--", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "**Innovative approach to the relaxation time problem.** The authors address the increased relaxation time in deeper networks indirectly by effectively reducing the depth over which error signals propagate. After several multiplications by the $\\beta_i$ coefficients that scale the feedback weights, negligible signal remains. The residual connections provide an intuitive mechanism for ensuring all layers receive teaching signals.", "weaknesses": "My main concerns are that the results are difficult to trust and the paper lacks proper analysis of what the method is actually doing.\n\n**Insufficient analysis of the method's mechanisms.** The proposed architectural modifications can be understood as a way to learn only layers that have direct impact on the output (similar to reservoir computing). This interpretation raises several critical questions: How far in the hierarchy are accurate error signals propagated? How would the proposed method behave if only the last layer or last few layers were trained? Answering these questions is essential for scientific understanding of the proposed modification and is currently absent. As a side note, the proposed modifications are rather independent to EP, analyzing their effects on other biologically plausible learning algorithms would strengthen the analysis.\n\n**Results are difficult to trust.** This stems from both presentation issues and technical inaccuracies:\n\n- The dynamics of Equation (2) are not energy-based dynamics and, more importantly, do not converge to stationary points of the energy function. Consequently, the EP update will only be an approximation. This limitation is not mentioned anywhere in the paper.\n- The EP framework (which receives minimal introduction) provides simple and elegant ways to describe the corresponding learning algorithm, yet the authors complicate it unnecessarily. For instance, they fix the error to its forward pass value; while correct, this adds unnecessary complexity. Algorithm 1 occupies more than half a page describing something explained concisely in the main text. Using the same symbol ($\\beta$) for both feedback strength and nudging strength creates confusion. While these could be dismissed as cosmetic details, collectively they create an impression of confusion that undermines the paper.\n- The typical baseline for EP is recurrent backpropagation, not standard backpropagation, since the network is recurrent. Backpropagation can only be applied to a feedforward version of the model, which is generally non-trivial to define. Consequently, the empirical comparisons feel like comparing apples to oranges.\n\n**Additional remarks:**\n\n- Figure 2 requires more statistically robust analysis.\n- It is unclear why the authors sweep over forward connection strengths in Section 4.1, as this appears unrelated to their method.\n- Figure 6 does not convincingly demonstrate that \"Larger β_i requires more iterations for the RNN to reach fixed point.\"\n- Empirical results are overly descriptive and sometimes lack proper interpretation. For example, the takeaway from Section 4.3 is unclear.\n\nIn my opinion, the paper is not currently ready for acceptance. It requires another extensive round of improvements to both the results themselves and their presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OKsZ9XtbcC", "reviewer": "Reviewer_7T1F", "summary": "This paper proposes an improved network architecture for training with equilibrium propagation. This new architecture includes scaling down of feedback signals to improve convergence speed and, to prevent vanishing gradients caused by the weaker feedback, residual connections in both forward and backward direction.\n\nThe contributions in this paper are indeed important steps in making equilibrium propagation more practically feasible, as it's applicability is often hampered by excessively long training times and the restriction to small networks.\nHowever, in the current state I can not recommend acceptance of this paper, for two main reasons. First, the presentation of the methods and results in the paper lacks clarity in many places (see details below in the \"Questions\" segment), which hampers the understanding and the applicability of the presented methods for others. Second, the experimental evidence is limited and a crucial sanity-check/ablation study is missing (see details below in the \"Weaknesses\" segment).\n\nIf these concerns are sufficiently addressed in the rebuttal, I am willing to increase my score of the paper.", "questions": "Areas of the paper lacking clarity:\n\n- Fig 1, Eq 2 and surrounding text contradicting each other:\n\t- The text around Eq 2, the surrounding inline equations and the referenced Fig 1 describe a 2-hidden-layer setting (explicitly mentioned in the text) but Eq 2 does not match that, it seems to describe a single layer setting. This is confusing.\n\t- The quantity alpha which is part of the model depicted in Fig 1 is missing in the Eq 2, it is not clear for which of the forward weights this scaling is really applied and for which it is not?\n- Recurrency in the network (Eqn 1, 2 vs Alg. 1):\n\t- Eqs 1 + 2 feature an explicit recurrency within the layer. In Eq 2 this is expressed via the parameter W. However, Alg. 1 and Fig 1 do not feature those weights.\n\t- What about the learning of the layer-internal weights W? Are they not adjusted? How are they chosen if they are fixed?\n\t- Do your networks contain layer-internal recurrent weights W or not?\n- Fig 1b: the depicted network structure is not helpful to actually understand the CNN-based setup. Could there be a more extensive illustration, e.g. in the appendix?\n- Fig 2:\n\t- What is the value denoted by color? The state variable s? If yes of which layer? A label on the color bar would help.\n\t- If two color bars for left and right columns would be used, the actual values would be better visible (the two columns have different ranges of values, so if each gets a colorbar more details are visible).\n\t- Caption: \"The hidden layer neurons are numbered from input to output.\" What does that mean? How can they be arranged, it's a recurrently connected set of neurons? What makes one closer to the input and another closer to the output?\n- Choices/description of experimental setup unclear:\n\t- l. 198-200: How exactly is the comparison with BP and FA done? Is the same network architecture trained with BP or FA (i.e. BP through time) or is a vanilla ANN of the same feed-forward sizes used?\n\t- Section 3.3: Why can't you have AGT for symmetric setups or regularly spaced residuals like in Fig3a for asymmetric setups?\n\t- Why do the numbers in Fig 5 not match the numbers in Tab 1?\n\t- Did the setups in Fig 4 + 5 have skip connections?\n\t- Tab1: Why are there no BP results for the 2HL case?\n\t- Tab2: According to the statement that T = 10 x N_hidden was used, the convergence times set for these results is much higher than anywhere else in the paper. Why? Also, this should be at least mentioned as a qualifier in the text.\n- Fig 6:\n\t- What is the network size (layer widths not given)?\n\t- What is K?\n\t- Caption: \"By default T = 10 * N_hidden\" Does that really mean that you are comparing networks of different sizes here? And that in a - d) the red network has a hidden layer that consists of 1 neuron? \n- Comparison of accuracies in Figures 4 and 6:\n\t- When accuracies are plotted for multiple different models it is very difficult to actually see the final accuracies (and their differences) in the plots. It would be much easier to see if errors instead of accuracies were plotted and the y-axis would be on a log-scale.\n\t- Fig 6: Additionally, it would be much easier to interpret, if the plots shared the same y-axis range.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- Making the dynamics in EP converge faster by scaling down the feedback signals is an important contribution, as the slow training times are a big drawback of the standard EP methods.\n- Adding the residual connection is a good and, as far as I am aware, novel idea. It directly tackles the problem of vanishing gradients/error signals in lower layers, which is made worse by the weak feedback used to speed up convergence. Therefore, the paper immediately provides a solution to a drawback of its proposed method.", "weaknesses": "Experimental evidence:\n- Networks with one hidden layer can solve MNIST. If a network with more hidden layers is trained on MNIST (or in fact even if the single hidden layer is wide enough), it can happen that only the weights of the top layer need to adjust to produce very high accuracies (> 95%). Therefore, just achieving high MNIST accuracies in a big (> 1 layer and > 200 neurons) does not actually prove that meaningful error signals are reaching the lower layers (which is the point of an algorithm aiming to approximate backprop). \n- I suggest the following (in my opinion crucial) ablation study to remove this doubt:\n\t- Train a network normally (like you did) and observe the resulting distributions of the weights in each layer.\n\t- Initialize a new network randomly according to those distributions.\n\t- Freeze the weights in the lower layers and train the others.\n\t- Compare accuracies. If accuracies reached in this ablated model are significantly lower than in the fully trained case, we can be convinced that meaningful error signals reach the lower layers.\n- The used networks have too many layers, more than are needed to solve the task. This is illustrated by the fact that also in the BP networks the most shallow ones have the best accuracy (even though they have the fewest parameters). Due to this heavy overparameterization I am not sure if any real conclusions can be drawn from these results, because the lower layers were never needed to actually solve the tasks: For example in Sec. 4.3 the paper concludes  \"Here, we found that due to gradient differences across different layers induced by weak feedback, a 3-hidden-layer RNN at βi = 0.01 (Table 1, ‘ours (tanh)’) learns well with a uniform learning rate\", i.e. the weight updates for the lowest layers do not need to be amplified by a higher learning rate. At the same time, the next section states that with increasing depth, gradients are weaker and weaker due to the attenuation via beta. An explanation in which the two statements do not contradict is: it did not matter if the lower layers only received a very weak error signal, because they were unnecessary in the first place.\n- The paper claims in prominent places (second sentence of the abstract, last point in the contributions and final sentence of the paper) to be a strong candidate for applicability on neuromorphic hardware and to be biologically plausible. In order to be able to claim this, in my opinion, it is necessary to prove that this algorithm deals well with noise (which is prevalent in both biological and neuromorphic systems). In particular it needs to be shown, that weight quantization and/or noise on weights as well as time-varying noise on the state variables s do not substantially impact functionality.\n\nLiterature:\n\n- Weak feedback has been a prominent feature in other bio-plausible backpropagation algorithms, this should be acknowledged [1, 2, 3]\n\n[1] Sacramento, João, et al. \"Dendritic cortical microcircuits approximate the backpropagation algorithm.\" _Advances in neural information processing systems_ 31 (2018).\n\n[2] Haider, Paul, et al. \"Latent equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons.\" _Advances in neural information processing systems_ 34 (2021): 17839-17851.\n\n[3] Meulemans, Alexander, et al. \"Credit assignment in neural networks through deep feedback control.\" _Advances in Neural Information Processing Systems_ 34 (2021): 4674-4687.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mwfb2Asug7", "reviewer": "Reviewer_iwsx", "summary": "This paper proposes a feedback-regulated residual recurrent neural network (FRE-RNN) to address the instability, slow convergence, and high costs of Equilibrium Propagation, a biologically plausible alternative to backpropagation that uses RNN dynamics for credit assignment. By scaling down feedback strength to accelerate settling and adding residual connections to counter vanishing gradients in deep networks, the approach draws from brain-like cortical modulation and recursive topologies. Evaluated on MNIST and CIFAR-10, FRE-RNN achieves BP-comparable accuracy with orders of magnitude speedups and enhanced stability.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- FRE-RNN shows practical improvements to EP while matching backpropagation accuracy on MNIST.\n- The paper includes systematic ablations on hyperparameters such as T, K, β_i, α_i, and network depth. Comparisons in Table 1 and Figure 6 illustrate key trade-offs. \n- This work advances energy-efficient computing suitable for neuromorphic hardwares.", "weaknesses": "- Individual components like feedback scaling and residual connections are not novel, and the biological framing is rather superficial.\n- Evaluations are limited to MNIST and CIFAR-10 datasets. There are no tests on sequential or natural language processing (NLP) tasks. Can the authors demonstrate the proposed method on at least one sequential processing / NLP tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)