Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism
Authors: Mete Erdogan
Cengiz Pehlevan
Alper Tunga Erdogan
Venue: NeurIPS 2025 spotlight ()
Keywords: Biologically Plausible Neural Networks
Backpropagation Alternative
Error Broadcasting
Direct Feedback Alignment
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose a principled error broadcasting framework to serve as a more biologically realistic and flexible alternative to the backpropagation algorithm, based on the orthogonality property of nonlinear MMSE estimators.

Abstract: We introduce *Error Broadcast and Decorrelation* (EBD), a novel learning framework for neural networks that addresses credit assignment by directly broadcasting output errors to individual layers, circumventing weight transport of backpropagation. EBD is rigorously grounded in the stochastic orthogonality property of Minimum Mean Square Error estimators. This fundamental principle states that the error of an optimal estimator is orthogonal to functions of the input. Guided by this insight, EBD defines layerwise loss functions that directly penalize correlations between layer activations and output errors, thereby establishing a principled foundation for error broadcasting. This theoretically sound mechanism  naturally leads to the experimentally observed three-factor learning rule and integrates with biologically plausible frameworks to enhance performance and plausibility. Numerical experiments demonstrate EBD’s competitive or better performance against other error-broadcast methods on benchmark datasets. Our findings establish EBD as an efficient, biologically plausible, and principled alternative for neural network training.

Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism
Mete Erdogan'2 Cengiz Pehlevan Alper T. Erdogan 1.2
KKUI Al Center, Koc University, Turkey EEE Department, Koc University, Turkey 'John A. Paulson School of Engineering & Applied Sciences, Harvard University, USA
KKempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, USA CCenter for Brain Science, Harvard University, USA
[merdogan18, alperdogan 0kk edu. cpehlevan@seas harvard, edu
Abstract
We introduce Error Broadcast and Decorrelation (EBD), novel learning framework for neural networks that addresses credit assignment by directly broadcasting output errors to individual layers, circumventing weight transport of backpropagation. EBD is rigorously grounded in the stochastic orthogonality property of Minimum Mean Square Error estimators. This fundamental principle states that the error of an optimal estimator is orthogonal to functions of the input. Guided by this insight, EBD defines layerwise loss functions that directly penalize correlations between layer activations and output errors, thereby establishing a principled foundation for error broadcasting. This theoretically sound mechanism naturally leads the experimentally observed three-factor learning rule and integrates with biologically plausible frameworks to enhance performance and plausibility. Numerical experiments demonstrate EBD's competitive better performance against other error-broadese broadcast methods on benchmark datasets. Our findings establish EBD as efficient, biologically plausible, and principled alternative for neural network training. The implementation is available at: https: httpc///eerrrraaaeeeeeeeerraaaaa
Introduction
Neural networks are dominant mathematical models for biological and artificial intelligence. A major challenge in these networks is determining how to adjust individual synaptic weights to optimize a global learning objective, known the credit assignment problem. In Artificial Neural Networks (ANNs), the most common solution the backpropagation (BP) algorithm [1]-
In contrast to ANNs, the mechanisms for credit assignment in biological neural networks remain poorly understood. While backpropagation is highly effective for training ANNs, it not directly applicable to biological systems because it relies biologically implausible assumptions. In its standard form, backpropagation propagates output errors backward through a separate pathway, reusing the same synaptic weights in the forward pass (Figure la). This requirement for weight symmetry is not supported by biological evidence [2]. Although many experimentally motivated models of local synaptic plasticity have been proposed [3], biologically feasible theory of credit assignment that integrates these mechanisms remains unresolved.
To address the credit assignment problem in biological networks, researchers have proposed methods known as error broadcasting [4-9]. These methods involve broadcasting the global output error directly to all layers, often through random projections fixed pathways, without relying on precise backward paths or symmetric weights (as summarized in Section 1.1), This eliminates the weight
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

symmetry issue inherent in backpropagation. Error broadcasting offers practical benefits for hardware implementation; recent work [10] demonstrates potential for efficient neural network execution. However, despite encouraging progress in both theory and application [11, 12], error broadcasting still needs stronger theoretical foundations to fully validate and enhance training effectiveness.
In this context, we introduce a novel learning framework termed the Error Broadcast and Decor- relation (EBD), which builds basic error broadcasting by introducing layer-specific objectíves grounded in estimation theory. The fundamental principle of EBD is to adjust network weights to minimize the correlation between broadcast output errors and the activations of each layer. This approach is rigorously grounded in Minimum Mean Square Error (MMSE) estimation, where an optimal estimator's error is orthogonal to any measurable function of its imput. We leverage this orthogonality principle for EBD, defining layer-specific training losses to drive layer activations (functions of the network input) towards orthogonality with the broadcast error. This enables more distributed mechanism for credit assignment, alternative to approaches relying solely on an output-defined loss and end-to-end error propagation.
Layer-1
Laye-1] lest
Leyen2 Tran
1
Test
Wx47
W
W
Avg. Abs. Cort. Coeft
0.05
Layer-3 Test
Layer3 Tran
0.06
22
2
0.04
Wx
W
(1)
(3)
6-
00
0.02
W
W
Epochs
(a) BP
(b) EBD
(c) Correlation between
layer activations and output error.
Figure 1: Comparison of error feedback mechanisms and correlation dynamics in multilayer percep- trons. (a) Backpropagation (BP) transmits errors sequentially through symmetric backward paths. (b) Error Broadcast and Decorrelation (EBD) broadcasts output errors to all layers using error-activation cross-correlations. (c) Average absolute correlation between layer activations and the output error during BP training on CIFAR-10 10 with MSE loss, illustrating its decline over epochs (see Appendix J).
EBD directly broadcasts output errors to layers, simplifying credit assignment and enabling parallel synaptic updates. It offers two key advantages for biologically realistic networks. First, optimizing EBD's loss naturally leads to experimentally observed three-factor learning rules [13, 14), which extend Hebbian plasticity by incorporating a neuromodulatory signal (the third factor) modulating synaptic updates based pre- and postsynaptic activity. Second, by broadcasting errors directly to layers as shown in Figure lb, it overcomes the weight transport problem inherent in backpropagation and some more biologically plausible credit assignment approaches [15, 16].
We demonstrate EBD's utility by applying to both artificial and biologically realistic neural networks. Benchmark results show EBD matching/exceeding ofte-offthe-ar error- broadcast techniques. Its successful application to a 10-layer biologically plausible network (CorInfoMax-EBD) provides initial evidence of depth scalability for more complex tasks.
1.1 Related work and contributions
Several frameworks have been proposed as alternatives to the backpropagation algorithm for modeling
credit assignment in biological networks [8]. These include predictive coding [15, 17, 18], similarity matching [16, 19], time-contrastive approaches [20-22], forward-only methods [23-25], target

---

propagation [26-28], random feedback alignment [29], and learned feedback weights [30, 31]. Alternative strategies also seek establish local learning rules by optimizing statistical objectives. such as the Hilbert-Schhe Schhett Independence Criterion (HSIC) bottleneck [32].
Another significant alternative is error-broadeast methods, where output emrors are directly transmitted to network layers without relying on precise backward pathways or symmetric weights. Two important examples of this approach are weight and node perturbation algorithms [4, 33-35J, in which global error signals are broadcast to all network units. These signals reflect the change in overall error caused by individual perturbations in the network's 's weights or units. more recent and prominent example of eπrоr broadcast is Direct Feedback Alignment (DFA) [6]. In DFA, the output errors are projected onto the hidden layers through fixed random weights, effectively replacing the symmetric backward weights required in traditional backpropagation. The core challenge of weight transport has been tackled by several other methods, many of which also rely on fixed random signals or avoid feedback entirely [36, 37]. Encouragingly, a number of these demonstrated the ability to scale effectively to large datasets, underscoring their potential as viable modification to the feedback alignment approach (which replaced the symmetric weights of the backpropagation algorithm with random ones). DFA has been extended and analyzed in several studies [11, 12, 39-41], demonstrating its potential in training neural networks with less biologically implausible mechanisms. Clark et al. [9] introduced another broadcast approach for network with vector units and nonnegative weights for which three factor learning based update rule is applied.
seggiccallyplausibee frameworks have
training mechanisms [38]. This approach first emerged
Our framework for error broadcasting differentiates itself through
a principled method based on the orthogonality property of nonlinear MMSE estimators,
eтror projection weights determined by the cross-correlation between the output errors and the layer activations as opposed to random weights of DFA,
dynamic Hebbian updating of projection weights as opposed to fixed weights of DFA,
updates involving arbitrary nonlinear functions of layer activities, encompassing a family of three-factor learning rules,
the option to project layer activities forward to the output layer.
In summary, our approach provides theoretical grounding for the error broadcasting mechanism and suggests ways to enhance its effectiveness in training networks.
2 Error Broadcast and Decorrelation method
2.1 Problem statement
To illustrate our approach, we first assume multi-layer perceptron (MLP) network with L layers. We label the input x (0) g and layer activations (k) g- for k: I,..., .....,,, L, where (k) is layer size. The layer activations are:
h (k) = ((() (1)
(k) w(k) + (c))
(I)
where k E {1,.. L} is the layer index. f(k) are activation functions, (kk) weights, (k) preactiva- tions and (kk) biases. We consider input-output pairs (x, sampled from a joint distribution P(x, y). The performance criterion is the mean square of the output error e = (2) -y, i.e. [1
2.2 Error Broadcast and Decorrelation loss functions
To guide the training of our neural network (which aims to minimize this MSE), we draw inspiration from the fundamental principles of Minimum Mean Square Error (MMSE) estimation theory [42]. This theory defines ideal estimator, denoted y. (x), which achieves the absolute minimum possible MSE for a given joint data distribution P(x, y). A crucial characteristic of this optimal estimator is its stochastic orthogonality property, which forms the theoretical cornerstone of our EBD approach.
Formally, considering input-output pairs (x, y) drawn from P(x, y). this optimal nonlinear MMSE estimator is given by yx(x) = [xy| (its derivation and properties the optimal USE-minimizing
3

---

function are detailed in Appendix A, Lemma A.1). Its estimation error = (x) satisfies:
Elg(x)<))=0.
for any properly measurable function g(x) of the input x (see Appendix A, Lemma A.2). This means 6, is orthogonal to g(x) (i.e., their expected outer product is zero). While this orthogonality property, stated in Eq. (2), foundational, its application in constructing estimators has predominantly been in linear MMSE estimation. In that context, the estimator yxx) is constrained to be a linear function of x, and under the linearity constraint on the estimator, Eq. (2) is restricted to a form where g(x) = x. This restricted orthogonality condition has long been used to derive parameters for linear estimators, such as Wiener- Kolmogorov and Kalman filters [43].
A key aspect of our work is to leverage the full generality of the orthogonality condition Eq. (2) for obtaining nonlinear estimators. For such estimators, this condition holds for any measurable function g(x) and, crucially, is not only necessary but also sufficient for MMSE optimality (as established in Appendix B, Theorem B. EBD distinctively employs this sufficiency as a constructive principle to train nonlinear estimators, specifically the neural network parameters.
We model the neural network (Eq. (1)) as a parameterized nonlinear estimator fe(x) = h (2) (x: +) and aim satisfy Eq. (2) for its output error fe (x) y. We choose g(x) as the network's 's hidden layer activations h (k) (x:; (())), where (k) (W(k), b (()). This choice is motivated because:
(i). Since each hidden-layer activation a nonlinear function of input x, the output error of optimal estimator should be stochastically orthogonal to those activations. Figure 1c illustrates this phenomenon by showing the evolution of the average absolute correlation between layer activations and the error signal during backpropagation training of an MLP with three hidden layers on the CIFAR-10 dataset, based on the MSE criterion. Similar correlation declining trends are also observed across different datasets and architectures (see Appendix J). The declining correlation during MSE training reflects the MMSE estimator's stochastic orthogonality of layer activations and output errors..
(ii). (*)) depends layer parameters (*), enabling their direct updates via differentiation.
(iii). if hidden-layer activations form a "rich enough" set of functions of x (as elaborated below), then enforcing error orthogonality to them implies orthogonality to "every" function of x.
Indeed, in Theorem B.2 of Appendix B, we show that when the hidden-layer activations-say those in the first fayer-form sufficiently rich basis (for example, becoming dense in L2(Px) as network width tends to infinity), enforcing that the output error e be orthogonal to these activations naturally drives the estimator toward the true MMSE solution. Accordingly, we aim to enforce zero correlation between the output error E and hidden layer activations, or more generally their nonlinear functions:
R (8)
Elg (b) (8))
0, h: = 0, .., T, with the typical choice (b))
h (R)
Building the orthogonality property and its established sufficiency for optimality (Appendix B. Theorem B.I), we define layer-specific surrogate loss functions that enforce orthogonality conditions with respect the hidden layer activations. As demonstrated in Section 2.3, these losses yield an alternative to backpropagation by broadcasting output errors directly to network nodes (Figure lb).
Specifically, based on the stochastic orthogonality condition in Eq. (3), we propose minimizing the Frobenius norm of the cross-correlation matrices R&(e) as replacement for the standard MSE loss. To this end, we define the estimated cross-correlation matrix between function and the output error for batch m and layer k as
(8) of layer activations
R (k) [m]] = AR (k) ge
ge (m - 1] 1 (K)) (m) [mm]""
B
where m is the batch index, X E [0, 1] is the forgetting factor used in the autoregressive estimation, B is the batch size, (2) [0] is the initial value hyperparameter for the correlation matrix, and + B)) ].
(k) mm] = g (k) (+) mmB + +])
g
(4)
the matrix of nonlinearly transformed activations of layer for batch m. In the above equation, mB + refers to absolute (sequence) index for the 1th member of batch-m. Furthermore,
Emm] c(mB +1]
c.mB B] ],
(5)

---

is the output error matrix for batch m. We then define the layer-specific loss function based on the stochastic orthogonality condition for layer k
Jom (K) mm) 2
R (2) ge
m
(6)
F
where I denotes the Frobenius norm. This loss function captures the sum of the squared magnitudes all cross-correlations between the components of the output error and the (potentially transformed) activations of layer k. Thus, we refer to the minimization of this loss as decorrelation.
2.3 Error Broadcast and Decorrelation algorithm
The functions in Eq. (6) defines individual loss functions for each hidden layer, which are used to adjust the layer parameters. These loss functions can be minimized using a gradient based algorithm.
To minimize the loss for layer k. we compute the gradient of the loss function ((k) (h (R), with
respect to the weight (R) The derivative can be decomposed into two terms:
is"
(
aG [m] aw (k) 13
2E m ow
aw!h (k) [m] CT'
R [m]] [m
+ +CTr
R ge
[m]
G
[m]
(R)
[m]lo
[Aw] [m])
where Ç := (1- -->/B. Similarly, the derivative with respect to the bias ((h)
is given by:
()(()) (k),
aG
(k)
[m]]]
E ml
ab [m] = CTr
(R) ge
[m]
[[m]
+ CTr
R (k) Re
[m]
(*) [m]]
ab.
ab (R)
[ab (Ab())mm [m]]
[Ab]
[m]]
Here ( Abi [m] (A Ab (m)) represent the components of the gradients contain-
(8)
2
ing derivatives of activations (output errors) with respect the layer parameters. As derived in Appendix C.I, we obtain the closed-fome expressions for Aw(k)
[m] and Ab;
(a) [m]:
[m+1]B Σ
(m+1)B
A AW [m]] =
(n)))) [m).
Ab (k) [m]] =
Σ (00)
(7)
B+1
where [n] g
(/(() (n. (3)) [m]) (())) (u. [m)] (k) [n]. (h and (*()) denote the derivatives of the
nonlinearity (k) and the activation function (k), respectively. The term q [m] defined as:
q (k) [m]
R
ge [m] E(ma)
representing the projection of the output error onto the layer activations, with the cross-correlation matrix R&ge [m]] as the transformation matrix. These projections are shown in Figure 1b.
For the special case of batchsize, B 1, the weight update in (7) simplifies to (8) (k) ; (h [m]) ((0) (u. [m]))! [m]h (k) (k- (k-) [m].
(AW!k)
(k) [m]les = Cg
[m]]
(8)
The update terms A (w [m] and Ab(a) [m] aim to adjust the activations to gradually become orthogonal to e, as they are based on the derivatives of activations with respect to layer parameters.
In contrast, A (W) [m] and (1) derived from the derivatives of the output error, work to push the output errors into a configuration more orthogonal to the activations. While both update
types strive for decorrelation, a critical distinction exists: A AwCR (k)
[m] and (b(4) [m] depend only
on layer activations and broadcast error signals, whereas A Aw, (k) [m]] and Ab, (k) [m] rely on signals propagated backward from the output layer, resembling backpropagation (see Appendix C.2).
By focusing solely on A (k) ing in a completely localized update mechanism for training the neural network. This simplification to
[m] and Ab(a) [m], we eliminate the need for propagation terms, result-
5

---

localized updates is supported by their positive alignment with backpropagation and full (untruncated) EBD gradient directions, as demonstrated in Appendix E.1 and E.2. respectively. Therefore, we prescribe the Error Broadcast and Decorrelation (EBD) update expressions as:
W mm + 1] W [m] (m]] A (8) [m]. (m0 + 1| = b (m0(m) (b(²) [m] Ab; (8)
[m],
for k = I,...,-- 1, where μ [m]] is the learning rate for layer & at batch m. Although these updates resemble backpropagation, key difference lies in the error signals: the backpropagated error is replaced by the broadcasted error. Furthermore, the algorithm introduces flexibility by allowing the choice of nonlinearity functions gk)), which influence the gradient terms A ! [m]] and Ab; [m]]
For the final layer (k = L), we utilize the standard MMSE gradient update:
11 [m]]
(m++ L)B
w mm + 1] - [m]
Σ
(p(((()())))) @ e(ril) h h(4-1)
B
n=mB+1
1 [m]
(m++1)B
b(D) m + 1] =b(") [m]
Σ
galh)
(n @ c[n],
B
R=mB+1
where p((t)) the derivative of the activation function of the output layer.
2.4 Further EBD algorithm extensions
We propose further extensions to the EBD framework to address potential activation collapse. which can arise when minimizing correlations is the sole objective. To prevent unit-level collapse, we introduce power regularization, while entropy regularization is employed collapse. Both regularizations can be implemented in ANNs as well as biologically plausible networks. The biological plausibility of employing these regularizers in MLP--baed EBD discussed in Appendix H.1. Although CorInfoMax- EBD inherently includes entropy regularization, it can also benefit from the addition of power regularization for enhanced stability. Additionally, we introduce forward layer activation projections improve the algorithm's versatility. We also extend the EBD formulations to more complex architectures, including Convolutional Neural Networks (CNNs) and Locally Connected (LC) networks. For further details on these extensions, please refer to Appendix D.
prevent dimensional
2.4.1 Avoiding collapse
A critical challenge with EBD is potential activation collapse, where decorrelation losses (Eq. (6) minimized by driving activations h 0, even with non-zero output errors, undermining learning. To counteract this, we introduce two complementary regularizers:
Power normalization: To prevent total activation collapse, power normalization (Eq. (9)) regulates layer activation power around a target level p(k)
(A)
(m+1)B
P
(h (8)
[m]] = Σ
(
Σ
h
(k)
[n] - p()))2
(9)
1=1
n=m B+1
which simplifies
N(
J (h(+))
= Σ 1=1
(1g (1) [m) - p(())22
(10)
for B = 1.
Layer entropy: To mitigate collapse into low-dimensional subspaces, which restricts expressiveness. we incorporate layer entropy (Eq. (11)), building on prior work [44, 45].
J (h (8) [m] = log (R (k), [m] + (*)).
(11)

---

Here, R (8) [m]] is the layer autocorrelation matrix, updated autoregressively with forgetting factor Ag: (k) i 1] + (1 - Xx)
R [m]] = >
(m [m] [m]""
(12)
B
where
H [m]] h (d) mB 1] ... h [mm 1) ]
(13)
the activation matrix. Gradient derivations for these objectives are Appendices C.3 and C.4.
2.4.2 Forward broadcast
In the EBD algorithm (Section 2.3), output errors are broadcast to layers to adjust weights and reduce correlations with activations. To complement this, introduce forward broadcasting, projecting hidden layer activations onto the output layer to optimize the decorrelation loss by adjusting the final layer's parameters. Details are provided in Appendix C.S.
2.4.3 Extensions to other network architectures
The EBD approach is independent of network topology. We extend EBD to convolutional neural networks (CNNs) in Appendix D.1 and to locally connected (LC) networks in Appendix D.2.
3 EBD for biologically realistic networks
In the previous section, we introduced the EBD algorithm within the context of MLP networks. While MLPs can resemble biologically plausible networks depending on the credit assignment mechanism, this section, we extend the application of the EBD approach neural networks that exhibit more biologically realistic dynamics and architectures, motivated by its inherent solutions to key neuroscientific challenges: 1) EBD's direct error broadcast naturally resolves the problematic weight symmetry requirement of BP, and 2) its update rules intrinsically manifest modulated, extended Hebbian mechanisms (three-factor learning), aligning with current understanding of synaptic plasticity. In explore how EBD relates to the biologically plausible three-factor learning rule and demonstrate its integration with the biologically more realistic CorInfoMax networks [45].
Pre-synaptic
Post-syuaptic
Output Error
Synapse
(5)
A AWAA oz 1/--1) "0.0 (f
Figure 2: three-factor synaptic update. The presynap- firing rate (/ft-)) (green, left) projects onto
Error-broadeast learning as
postsynaptic neuron 11(h) (blue, centre) through synapse (((k) A layer-specific broadcast of the (k) (yellow, right) provides the
output error q; modulatory third factor that gates plasticity. Together, presynaptic activity, postsynaptic non-linear deriva- tives 9;
the following subsections,
(() fr(k) (blue rectangle), and the modulatory
signal form the product that drives the EBD weight change AW displayed underneath the circuit.
3.1 Three factor learning rule and EBD
The three-factor learning rule for biological neural networks extends the traditional two-factor Hebbian rule by incorporating a modulatory signal into synaptic updates based on presynaptic and postsynaptic activity [13, 46]. While backpropagation can be expressed similarly, is not typically considered a three-factor rule in neuroscience, as its 'third factor' is locally tailored signal specific to each neuron requiring biologically implausible dual network with symmetric weights, unlike global neuromodulatory signals [13]. In contrast, EBD update for batchsize B in (8) naturally matches the three-fate factor structure:
(W ij
x g (k)
(1) (1) p((()) ( (up
Postsympticc
Modulatory Presynaptic
where (k) is the projected global enror. Thus, EBD supports various three-factor rules depending on nonlinearity (k), For example, gi (8)( (1(()) h (
yields the enrormodulated Hebbian update

---

[11, 47]: AW & (/ (())(() (2) (K) hnb-1), By enabling diverse three-factor updates via different nonlinear functions, EBD holds potential for modeling biologically consistent neural learning processes.
9 hi
ij
Figure breaks the EBD weight update (8) into its three interacting factors. The presynaptic activity h2 (k-1) from the sending unit; the postsynaptic term is (k) fr(k) computed from the receiving unit's own activation; and the modulatory broadcast error (8) derived from the network's output error Multiplying these three quantities produces the weight change AW shown beneath the diagram, revealing that EBD naturally realises the classical three-factor learning rule in neural networks.
3.2 CorInfoMax-EBII CorInfoMax with three factor learning rule
One of the significant advantages of the EBD framework is its flexibility broadcast output errors into network nodes, which can be leveraged to transform time-contrastive, biologically plausible approaches into non-contrastive forms. To illustrate this property, we propose modification of the recently introduced CorlnfoMax framework [45] (see Appendix F for summary). The CorInfoMax approach uses correlative information flow between layers its objective function:
[cr[m]] (i( (h (k-1) (1)))) [m] + 1/55>) (h
(h .h ()++))
where,
i(ex) (h( (8), (1+1)) [m]] = 0.5 log det (x,) [m] + =xd) - 0.5 log det det R 2a+ [m] ] + ExJ),
f(ew) (h (8), h ((++1)) [m]] = 0.5 log del (R [m] + e I) 0.5 log del ( [m]] + ExT),
are alternative forms of correlative mutual information between nodes, defined in terms of the correlation matrices of layer activations, i.e., R and forward and backward prediction errors (Ra and R.) Here, forward/backward prediction errors are defined by
(k+1) [n] = h (1(ht)(nn) - (( [m]] (*) [n],
(8)
(k) [n] = h (k0 (bb.)) [m]] (k+1) [n],
respectively. Here, W [m] (W((..) (m)) is the forward (backward) prediction matrix for layer
This objective leads to network dynamics corresponding to a structure with feedforward and feedback (A) that maximize layer entropy. In the original work [45], the two-phase EP approach [22] is proposed to train the network weights. As alternative, we propose employing the EBD update rule to replace the two-phase EP adaptation. The proposed EBDD algorithm is described by the following update equations defined in Algorithm 1.
prediction weights, and lateral connections
Algorithm 1 CorlnfoMax-EBi Algorithm for Updating Weights in Layer k.
Input: Batch size B, layer index k. iteration step learning rates ((ffA)) (((,)) μ (d) μ (dd.))) (dd,,))
factors Ad, Åg YE, activations (A) in Eq. (13), the nonlinear function of layer activations ((k) in Eq. (4) and their derivatives G()) in Eq. (28), the derivative of activations F(A) in Eq. (29), output error E in Eq. (5),
((k) ((k)
prediction errors E and E in Eq. (54)-(55), lateral outputs ( in Eq. (56).
Output: Updated weights W())k), (0..). ((k)
Step 1: Update error projection weights: R100) [m] AaR 1] م G [m][[m]
Step 2: Project errors to layer k: Q(0) [m] Rgg [m] E/m]
Step 3: Find the gradient of the nonlinear function of activations for layer A:
c (+0 [m]] = [m]] @ Q @ ( [m]]
Step 4: Update forward, backward and lateral weights for layer Az
[((+ (( [m 1] +
B
[rrc]
E
[m]]
B a (0 (+)] H (mm""
(k)
(+h [m] (+.))) [m 1] + B-1 (2:
E
[m]] B a (0. (m क i [ml] H (x (*+ [mr]"
B (+) [m) Ag B (m) - B- 1 7 2 [m]
[m]]
B 1 (1)))) Imm (ه |m] (+) [m]]
8

---

Table Accuracy (%) results for MLP, CNN, and LC networks on MNIST and CIFAR-10. Best and second besttd-best results are bold and underlined. GEVB results are from Clark et al. [9].
Dataset
Model
DFA
DFA+E (ours)
NN-
MS-
BP
EBD
GEVB
GEVB
(MSE)
(ours)
MLP
98.1+:0.1
98.24.0.33
98.1
97.7
98.7
98.2+10.08
MNIST
CNN
99.1+0.05
99.1+0.07
97.7
98.2
99.5+1.004
99.1=0007
LC
98.9
98.911.04
98.2
98.2
99.110.04
98.9-10.04
MLP
3..1
52.2:0.49
52.4
51.1
56.410.33
55.5=t0.19
CIFAR-10
CNN
58.4+1.59
58.6:10.66
66.3
61.6
75.2:.0.22
66.410.43
LC
62.2+0.21
62.1
58.9
59.9
67 0
64.3=+026
Table 2: Accuracy (%) results for the CNN on CIFAR-100.
Dataset
Model
DFA
BP (CE)
EBD (ours)
CIFAR-100
CNN
41.9++0.2
60.5--0.17
45.94-0.69
Here, we assume layer activations H()) forward (backward) prediction errors E (E), output error E, and lateral weight outputs Z are computed via CorInfoMax network dynamics in [45] (see Appendix F). Integrating EBD enables single-phase updates per input, eliminating the less biologically plausible two-phase mechanism required by CorInfoMax-EP EP's two-phase approach-separate label-free and label-connected phases is implausible, as biological neurons unlikely alternate between distinct global phases for learning. Our method simplifies the process, aligns more closely with biological learning, and achieves comparable or superior performance to CorInfoMax-EB (see Section 4).
The CorInfoMaxxE.B! scheme introduced in this section is more biologically plausible than the earlier based EBD formulation, as its learning rules can be implemented through local mechanisms such as lateral and three-factor Hebbian/anti-Hobbial updates, realistic neuron models with apical and basal dendrites, and feedback via backward predictors. Additionally, both the entropy and power normalization terms in CorlnfoMax are realizable using biologically plausible operations, particularly in the online setting with single-sample updates. See Appendix H.2 for further discussion.
Numerical experiments
In this section, we evaluate the performance of the proposed Error Broadcast and Decorrelation (EBD) approach on benchmark datasets: MNIST [48] and 11FAR-10/100) [49]. For experiments with MNIST and CIFAR-10 involving MLP, CNN and LC, we use the same architectures used in [9]; while for CIFAR- 100 we adopt a CNN architecture closely following that of [41]. We also tested the proposed CorInfoMax-EBD model against the CorInfoMax-EP model of [45]. More details about architectures, implementations, hyperparameter selections, and experimental outputs are provided the Appendix I.
EBD test accuracy results compared to BP (with MSE criterion) and three error- broadebr methods: DFA without and with entropy regularization (DFA-B) [6], global error vector broadcasting are in Table 1 for both MNIST and CIFAR-10. Under our training setup, BP yielded comparable test accuracies for these datasets with both MSE and Cross-Entropy losses, though we report only the MSE results. In addition, CIFAR-100 results of EBD compared to BP (with Cross- cooos-Entrrpy criterion) and DFA is given in Table 2. Lastly, the test accuracies for biological CorInfoMax networks trained with EP and EBD methods are in Table 3.
-NN-e-gaivve(NN-GEVV)) and
These results show that EBD-trained networks achieve equivalent performance on the MNIST dataset and significantly better performance on the CIFAR- 10 and CIFAR- 100 datasets compared to other error broadcasting methods. These improvements of EBD Table over DFA can be attributed to the adaptability of error projection weights in EBD. The improvement of CorInfoMax MarnfoMax-EBBD over CorInfoMax-EZ in Table can be attributed to CorInfoMax- EBD incorporating error decorrelation in updating lateral weights, whereas CarInfoMax-EP relies only on (abbia) updates. Particularly noteworthy is the performance of CorDnfoMax-EBBD, which not only substantially improves upon the original CorInfoMax-EI 20) but also demonstrates encouraging scalability with depth, with a 10-layer CorInfoMax-EDI achieving 96.38% on MNIST and 54.89% on CIFAR-10 10 using online learning (batch size 1). This highlights EBD's 's potential in deeper, more complex biological networks.
10 (e.g., 75%79% vs. 50.97% for 3-layers with batch size

---

Table 3: Accuracy (%) results for EP and EBD CorInfoMax (CIM) algorithms on MNIST and CIFAR-
10. Best and second- best are bold and underlined. Column marked with [*] is from Bozkurt et al. [45].
CIM-EP [*]
CIM-EBD (Ours)
CIM-EBD (Ours)
CIM-EBD (Ours)
Dataset
3-Layers
3-Layers
3 3-Layers
10-Layers
(batch size 20)
(batch size : 20)
(batch size I)
(batch size : 1)
MNIST
97.6
97.5++0.12
94.9++0.16
96.4+0..1
CIFAR-10
51.0
55.7+-0.17
53.4+0.33
54.9+-..55
5 Conclusions, extensions and limitations
Conclusions. We introduced the Error Broadcast and Decorrelation framework, a biologically plausible alternative backpropagation. EBD addresses the credit assignment problem by minimizing correlations between layer activations and output errors, offering fresh insights into biologically realistic learning. This approach provides theoretical foundation for existing error broadcast mechanisms and three-factor learning rules in biological neural networks and facilitates flexible implementations in neuromorphic and artificial neural systems. EBD's brood-broadcstting mechanism aligns with biological processes using local updates, and notably, has proven effective for training deep recurrent peglooccally-pluusbbee networks (c.g.. the 10-layer CorInfoMax-ENI -BD), thereby addressing key challenge in effectively scaling deep. biologically plausible learning with local rules. Moreover, EBD's simplicity and parallelism suit efficient hardware, like neuromorphic systems.
Extensions. The MMSE orthogonality property underlying EBD offers significant promise for new algorithms, deeper theoretical understanding, and neural network analysis in both artificial and bio- logical contexts. Further theoretical extensions, drawing from the groundwork laid in Appendix B.2, could focus on deriving tighter convergence guarantees for EBD in practical (finite-width) settings and on investigating the impact of more adaptive choices for the decorrelation functions (k)), In addition, EBD provides theoretical underpinnings for error-broadcast mechanisms with three-factor learning rules, enabling the conversion of two-phase contrastive methods into single-phase approach. We are currently unaware of similar theoretical properties for alternative loss functions. Finally, our numerical experiments in Appendix J.2 reveal that similar decorrelation behavior occurs for networks trained with backpropagation and categorical cross entropy loss, suggesting that decorrelation may be a general feature of the learning process and intriguing avenue for further investigation.
Impact and limitations. This paper seeks to advance the fields of Machine Learning and Computa- tional Neuroscience by proposing novel learning mechanism. As a foundational learning algorithm, we do not identify specific negative societal impacts arising directly from the EBD mechanisms beyond general considerations common to advancements machine learning. While EBD offers a pnporetically-groundee framework for error broadcast based and three factor learning that has yielded competitive (and in some cases, superior) performance against other error- broodbbo methods on the presented benchmarks, several aspects warrant future investigation:
Scalability: The current work evaluates EBD on MLP, CNN, LC and recurrent biological networks for image classification tasks like MNIST and CIFAR-10. Results the 10-layer CorInfoMax- EBD demonstrate the potential to scale EBD to deeper, biologically realistic recurrent architectures using online learning. However, assessing EBD's ' performance significantly larger datasets, or its applicability to diverse large-scale architectures in other domains, remains an important open direction. While related methods like DFA have been explored in such contexts [41], comprehensive empirical validation of EBD itself under those conditions is needed.
(k) Computational Cost and Hyperparameters: The dynamic updating of enror projection matrices Rgge and the optional inclusion of regularization terms like layer entropy (discussed in Appendix G and Appendix I.7) contribute to computational and memory overhead compared simpler schemes like DFA with fixed projectors, or standard backpropagation. EBD also introduces several hyperparameters (e.g.. learning rates for decorrelation and regularization, forgetting factors) that require careful tuning. although this offers flexibility. Future work could explore more efficient update mechanisms or automated tuning strategies.
10

---

Acknowledgements
This work was supported by KUIS AI Center Research Award. C.P. was supported by an NSF CAREER Award (IS-22339780) and Sloan Research Fellowship. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
[1] David E. Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning representations by back-propagating errors. Nature, 323(6888:)333333323 1986. doi: 10. 00.1038/323533a.
[2] Francis Crick. The recent excitement about neural networks. Nature, 337:129-132, 1989.
[3] Jeffrey C Magee and Christine Grienberger. Synaptic plasticity forms and functions. Annual Review of Neuroscience, 43(1):95-117, 2020.
[4] Ronald J Williams. Simple statistical ment learning. Machine Learning, 8:229-256, 1992.
algorithms for connectionist reinforce-
[5] Justin Werfel, Xiaohui Xie, and H Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Advances in Neural Information Processing Systems, 16, 2003.
[6] Arild Nokland. Direct feedback alignment provides learning deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[7] Pierre Baldi, Peter Sadowski, and Zhiqin Lu. Learning in the machine: Random backpropagation and the deep learning channel. Artificial Intelligence, 260:1-35, July 2018. doi: 10.10) 1016//.artint 2017.00 7706.000.3
[8] James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences, 23(3):235 250, 2019.
[9] David Clark, L.F. Abbott, and Sue Yeon Chung. Credit assignment through broadcasting a global error vector. In Advances in Neural Information Processing Systems, 2021.
[10] Ziao Wang, Kilian Müller, Matthew Filipovich, Julien Launay, Ruben Ohana, Gustave Pariente, Safa Mokaadi, Charles Brossollet, Fabien Moreau, Alessandro Cappelli, et al. Optical training of large-scale transformers and deep neural networks with direct feedback alignment. arXiv preprint wxXiv:2409.12965 2024.
[11] Blake Bordelon and Cengiz Pehlevan. The influence of learning rule representation dynamics wide neural networks. International Conference on Learning Representutions, 2022.
[12] Julien Launay, Iacopo Poli, and Florent Krzakala. Principled training of neural networks with direct feedback alignment. arXiv preprint arXiv:1906.04554 2019.
[13] Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Comeil, and Johanni Brea. Eli- gibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules. Frontiers in Newral Circuits, 12:53, 2018.
[14] Eukasz Kuśmierz, Takuya Isomura, and Taro Toyoizumi. Learning with three factors: modulat- ing hebbian plasticity with emrors. Current Opinion Neurobiology, 46:170-177.2011
[15] James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algo- rithm in a predictive coding network with local hebbian synaptic plasticity. Neural Computation, 1229 2017.
[16] Shanshan Qin, Nayantara Mudur, and Cengiz Pehlevan. Contrastive similarity matching for supervised learning. Neural Computation, 33(5):1300-1328 2021.

---

[17] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-fieee field effects. Nature neuroscience, 211)799-87. 1999.
[18] Siavash Golkar, Tiberiu Tesileanu, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. Constrained predictive coding as a biologically plausible model of the cortical hierarchy, Ad- vances in Neural Information Processing Systems, 35:14155-144 14169, 2022.
[19] Yanis Bahroun, Shagesh Sridharan, Atithi Acharya, Dmitri B Chklovskii, and Anirvan M Sengupta. Unlocking the potential of similarity matching: Scalability, supervision and pre- training. arXiv preprini urXiv:2308.02427, 2023.
[20] David H Ackley, Geoffrey E Hinton, and Terrence Sejnowski. learning algorithm for boltzmann machines. Cognitive Science, 91):1477-169 1985.
[21] Randall C O' O'Reilly. Biologically plausible error-driven learning using local activation differ- ences: The generalized recirculation algorithm. Neural Computation, 8(5):895-938, 1996.
[22] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11:24, 2017.
[23] Geoffrey Hinton.
The neward-forwarr algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13334. 2022.
[24] Matilde Tristany Farinha, Thomas Ortner, Giorgia Dellaferrera, Benjamin Grewe, and Angeliki Pantazi. Efficient biologically plausible adversarial training. arXiv preprint arXiv:2309. 2023.
[25] Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit assignment problem without backward pass. In International Conference Machine Learning, pages 4937-4955. PMLR, 2022.
[26] Yann Le Cun. Learning process in an asymmetric threshold network. In Disordered Systems and Biological Organization, pages 233-240. Springer, 1986.
[27] Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv: 7906, 2014.
[28) Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propa- gation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15, pages 498-515. Springer, 2015.
[29] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1): 2016.
[30] John F Kolen and Jordan B Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (/CNN'94) volume 3, pages 1375-1380. IEEE, 1994.
[31] Li Ji-An and Marcus K Benna. Deep learning without weight symmetry. arXiv preprint axXiv:2405.20594 2024.
[32] Wan-Duo Kurt Ma, J P Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back- propaaropag In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 5085-5092, 2020.
[33] Amir Dembo and Thomas Kailath. Model-free distributed learning. IEEE Transactions on Neural Networks, 1(1):58-70) 70, 1990.
[34] Gert Cauwenberghs. A fast stochastic error-descent algorithm for supervised learning and optimization. Advances in Neural Information Processing Systems, 5, 1992.
12

---

[35] Ila R Fiete and H Sebastian Seung. Gradient learning in spiking neural networks by dynamic perturbation of conductances. Physical Review Letters, 97(4)0048104 2006.
[36] Mohamed Akrout, Colin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. In Advances in Neural Information Processing Systems, volume 32, 2019.
[37] Corentin Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Fixed ran- dom learning signals allow for feedforward training of deep neural networks. Frontiers in Neuroscience, 15:629892, 2021.
[38] Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausii learning algorithms can scale to large datasets. In International Conference on Learning Representations (ICLR), 2019.
[39] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey Hinton, and Timothy Lillicrap. Assessing the scalability of poologicallymmotivate deep learning algorithms and architectures. Advances in Neural Information Processing Systems, 31, 2018.
[40] Donghyeon Han and Hoi-jun Yoo. Efficient convolutional neural network training with direct feedback alignment. arXiv preprint arXiv: arXiv:1901.01986 2019.
[41] Julien Launay, Iacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales to modern deep learning tasks and architectures. Advances in Neural Information Processing Systems, 33:9346 9360, 2020.
[42] Athanasios Papoulis and S Unnikrishna Pillai. Probability, Random Variables, and Stochastic Processes. McGraw-Hill Europe: New York, NY, USA, 2002.
[43] Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear estimation. Prentice- Hall information and system sciences series. Prentice Hall, 2000. ISBN 9780130224644.
[44] Serdar Ozsoy, Shadi Hamdan, Sercan O Arik, Deniz Yuret, and Alper T. Erdogan. Self- supervised learning with information maximization criterion. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[45] Bariscan Bozkurt, Cengiz Pehlevan, and Alper T. Erdogan. Correlative information maximiza- tion: biologically plausible approach supervised deep neural networks without weight symmetry. Advances in Neural Information Processing Systems, 37, 2023.
[46] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike-timing-dependee plasticity, and theory of three-factor learning rules. Frontiers in Neural Circuits, 9:85, 2016.
[47] Yonatan Loewenstein and H Sebastian Seung. Operant matching is generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proceedings of the National Academy of Sciences, (0344)):15224-152) 15229, 2006.
[48] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[49] Alex Krizhevsky. Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report Technical Report, University of Toronto, 2009.
[50] Anuran Makur. Inference and information lecture notes. Lecture Notes, 2017. URL
https://ww/ cs purdue. thomes/amakur/ 20Amuran//20aakur pdf. Purdue University, Department of Computer Science.
[51] Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing mini- mization with randomization in learning. In Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Léon Bottou, editors, Advances in Neural Information Processing Systems 21 (NIPS 2008). pages 1313-1320. Curran Associates, Inc., 2008.
13

---

[52] Ali Rahimi and Benjamin Recht. Uniform approximation of functions with random bases, In 2008 46th Annual Allerton Conference on Communication, Control, and Computing, pages 555-561. IEFE, 2008
[53] Yitong Sun, Anna C. Gilbert, and Ambuj Tewari. On the approximation properties of random ReLU features. arXiv preprint arXiv:1810.04374. 2018.
[54] Alex TL Leong, Russell Chan, Patrick Gao, Ying-Shing Sagg-Shing Chan, Kevin K Tsia, Wing-Ho Yung, and Ed X Wu. cong-range projections coordinate distributed brain-wide neural activity with specific spatiotemporal profile. Proceedings of the National Academy of Sciences, 113
(51):B8306-E8315. 2016.
[55] Leena Ali Ibrahim, Shuhan Huang, Marian Fernandez-Otero, Mia Sherer, Yanjie Qiu, Spurti Vemuri, Qing Xu, Robert Machold, Gabrielle Pouchelon, Bernardo Rudy, et al. Bottom-up inputs are required for establishment of top-down connectivity onto cortical layer neurogliaform cells. Neuron, (09(21):34733348) 2021.
[56] Yann LeCun and Corinna Cortes.
MNIST handwritten digit database.
http://yaamleecu..co//xxxbbbbbi
2010. URL http::/yame htc.l//annneee...mmmmmmmm lecun. cwm/ coom/exdbmnnis./
[57] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations (ICLR), San Diega, CA, USA, 2015.
[58] Tony F Chan, Gene H Golub, and Randall Le Veque. Updating formulae and pairwise algorithm for computing sample variances. In COMPSTAT' 1982 5th Symposium held at Toulouse 1982: Part I: Proceedings Computational Statistics, pages 30-41. Springer, 1982.
14

---

Appendix
Table of contents
Preliminaries on nonlinear Minimum Mean Square Error (MMSE) estimation
17
B On the stochastic orthogonality condition based network training
18
B.1 The stochastic orthogonality condition and linear MMSE estimation
18
B.2 The use of stochastic orthogonality conditions for nonlinear MMSE estimation
18
The derivation update terms
22
C.1 AW1 and Ábj calculation
22
C.2 A 22 and Ab; calculation
23
C,3 Update corresponding to the layer entropy regularization
C,4 Update corresponding to the power normalization regularization
25
C.5 On the EBD with forward projections
25
D Additional extensions of EBD
27
D.I Extensions to Convolutional Neural Networks (CNNs)
27
D.2 Extensions to Locally Connected (LC) Networks
29
Gradient alignment in EBD
E.1 Alignment between EBD updates and backpropagation gradients
31
E.2 On gradient truncation and biological plausibility
32
F Background on online Correlative Information Maximization (CorInfoMax) based biologically plausible neural networks
33
F.1
The derivation the CorInfoMax network.
33
F2
CorInfoMax-EP learning dynamics
34
F3
CorInfoMax-EPP
35
G Implementation complexity of the EBD approach
36
G.1
Complexity analysis: Error Propagation vs. Error Broadcast
36
H On the biologically plausible nature of Entropy and Power-normalizatioon updates
39
H.1 MLP implementation with Entropy and Power mowmaliormaiiztionn regularizations
39
H.2 CorInfoMax-FBB) implementation
39
H.3 Summary and conclusions
40
I
Supplementary on numerical experiments
41
I.1
Architectures
1.2
EorD
42
1.3 Multi-Layer Perceptron
46
15

---

1.4
Convolutional Neural Network
48
1.5
Locally Connected Network
50
1.6
Implementation details for Direct Feedback Alignment (DFA) and backpropagation
training
51
1.7
Runtime comparisons for the update rules
52
L8
Reproducibility
52
1,9
Computational resources
52
I.10
Accuracy and loss curves
53
J
Calculation of the correlation between layer activations and output error
56
J.1
Correlation in the mean squared error (MSH) criterion-based training.
56
J.2 Correlation in the cross-entropy criteiion-based training.
57
16

---

A Preliminaries on nonlinear Minimum Mean Square Error (MMSE) estimation
Let E IR and x E IR" represent two non-degenerate random vectors with a joint probability density function fyx(y. x) and conditional density fyix(y(x) The goal of nonlinear minimum mean square error (MMSE) estimation is to find an estimator function b I > RP that minimizes the mean squared error (MSE), which defined as:
MSE(b) Elly b())
Lemma A.1. The best nonlinear MMSE estimate of y given x is:
(x) - Ey)x(y/x))
The proof of Lemma A.I relies on the following fundamental result (see, for example, Papoulis and Pillai [42]), which is central to the development of the entire EBD framework in the current article:
Lemma A.2. The estimation error for b. (x) = yy |x], denoted as e, = y- (x),xx),, (x), is orthogonal to any vector-valued function g R - R of Formally, we have:
E Ecc.(x)'1 = 0.
Proof. (Lemma A.2)
The proof follows simple steps:
Ejc.g(x)"1 = Ex Eyix [(y -
Ex [[ysxly|xx] - (xyxxylx())g(x) 0.
Using Lemma A.2, we can now prove Lemma A.I:
Proof. (Lemma A.I) Let b R" + RRP be any arbitrary function. The corresponding MSE can be written as:
MSE(b) Billy b(x)
By adding and subtracting Ey [y xx, we can decompose the error as:
MSF(b) [|ly [x / b(x)
E - Eyix | 113)
+ B( By [y b
+ 2E - [y))) (Ey [y|x] b(x)))
Ellly Ey/x/y/>|1 H
(ylx)
b(x)
+ 2E [Tr ((y (
b(x))
= Ellly - /yy>(y/>>|13> E [x|]]
[y b(x)
+ 27 (E (By by b(x)) I))
The third term, representing the cross product, vanishes by Lemma A.2, leaving us with:
MSE(b). Ellly b. (x) + (x) b(x)
Since the second term always non-negative, the MSE minimized when b(x) bx(x).
17

---

B On the stochastic orthogonality condition based network training
This appendix provides further theoretical grounding for the Error Broadcast and Decorrelation (EBD) framework, addressing the use of the nonlinear Minimum Mean Square Error (MMSE) orthogonality condition for deriving estimators and the implications of orthogonality in nigh-dimensional spaces. We begin by re-establishing the geometric interpretation of stochastic orthogonality in the linear MMSE setting. We then demonstrate the sufficiency of the nonlinear MMSE orthogonality condition for estimator to be optimal. Crucially, we address, under reasonable assumptions, how the EBD algorithm, by enforcing orthogonality to hidden unit activations, can approximate orthogonality to arbitrary measurable functions the input, particularly in the infinite width limit. Finally, we briefly touch upon the scaling of these orthogonality conditions.
B.1 The stochastic orthogonality condition and linear MMSE estimation
Within the context of this article, stochastic orthogonality refers to statistical uncorrelatedness. The term orthogonality is defined within Hilbert space of random variables, where the inner product between two random variables and b is given by:
(a, b) & E[ab].
(14)
Here, the inner product is defined as the expected value of their product, which corresponds to their correlation. Two random variables and are said to be orthogonal their inner product is zero:
(a, b) = E(ab] = O.
(15)
Thus, two random variables are orthogonal if and only if their correlation is zero (see, for example, Kailath et al. [43], Chapter 3).
In the context of Minimum Mean Square Error (MMSE) estimation, where the goal is to estimate vector y Rº from observations IR" is known (e.g., Lemma A.2 in Appendix A) that the error e, of the optimal MMSE estimator (x) satisfies:
E [x.g()] 0 Opcks
(16)
where g R" B is an arbitrary vector-valued measurable function of x. This equation states that the cross-correlation matrix between the nonlinear function of the input g(x) and the output error a p x k zero matrix.
The matrix equation in Eq. (16) can be expressed more explicitly as p zero-correlation conditions:
E [er.a Øj (x)] = 0, i= I, 1, l. k.
(17)
Using the inner product definition in Eq. (14), we can rewrite Eq. (17) as p- k stochastic orthogonality conditions:
(eu,i, 9j (x)) 0, = j k.
(18)
This geometric interpretation, where correlation is viewed as inner product, foundational. In the linear MMSE estimation setting, the orthogonality condition in (18) is restricted by choosing g(x) to be linear functions of x, most commonly the identity mapping, i.c. g(x) This leads to:
(e,,,,z;) = 0, =
j = 1,..,,
(19)
These linear orthogonality conditions are fundamental and are used in reverse to derive the parameters of linear MMSE estimators, such as the Wiener and Kalman filters [43]. The estimator's structure is assumed (linear), and its parameters are found by enforcing these orthogonality conditions.
B.2
The use of stochastic orthogonality conditions for nonlinear MMSE estimation
The principle of using orthogonality conditions to define estimators extends to the nonlinear domain. key question is whether the nonlinear MMSE orthogonality condition can be similarly used "in reverse" to identify optimal nonlinear estimators.
18

---

B.2.1 The sufficiency of the nonlinear MMSE orthogonality condition
The nonlinear orthogonality principle states that for an optimal nonlinear MMSE estimator, the estimation error is orthogonal to any measurable function of the input. The following theorem establishes that this is not only necessary condition but also a sufficient one, providing strong theoretical basis for using these conditions to define and seek optimal estimators [50].
Theorem B.1 Nonlinear MMSE Estimation and Orthogonality Condition). Let y RP and Ik"" be random vectors with joint probability distribution. An estimator Ø(x) is the optimal nonlinear MMSE estimator, MSE (z), if and only if the error y - ((x) satisfies:
Ele ((x) $1 Opxt
for all measurable functiion R - TR for any 2 1.
Proof. The necessity part (-),, i.e., (x)) (MMSSF()) then the orthogonality condition holds, is a standard result (established, for instance, in Lemma A.2 Appendix A).
Here we prove the sufficiency part (-)) Let (e) be an estimator such that its error ee - (e) satisfics the orthogonality condition:
B((y - fe(x)) ==
(20)
for all measurable functions IR + k (where Kg is the dimension of 9(x)). Consider the difference between fx) and the true MMSE estimator (MMSE(*)) We can write:
(=) (u ÚмMSH(<)) + (ÜMMSH(()) [o()))
Substituting this into the assumed orthogonality condition for fe (z)):
E[((v - (MMSE<()) + (ÜMMSÆ(() - se(x))) g(x)" ] = Opxks
By linearity of expectation:
[yy ÜмMSE(*)) (((e))))))))))]))(((((((()))))))
E((MMSS(()) - fe(x)) g(x)
The first term is zero due to the (necessary) orthogonality property of the MMSE estimator (a). Therefore, we are left with:
[[(ÍMMSE(*)- - fe(x))
=
(21)
for any measurable function g(z). Choosing ((x) = ÚMMSE (z) fe (z), and taking the trace of Eq. (21), and applying the cyclic property of the trace operator, we obtain
E [[l[MMsE - (=) 0.
Since [[ - ( is non-negative random variable, its expectation being zero implies that [[)M)))(()) - fe(x)) =0 almost surely. This means ÚMMSF(*) fee (æ) almost surely, or fe(x) = (MMSE(() almost surely. Thus, fe is the optimal MMSE estimator.
This theorem provides a strong justification: if can find an estimator f=(x) whose error y- fe (a( is orthogonal to all measurable functions of æ, then fe(r) is indeed the optimal MMSE estimator. This underpins the EBD framework's objective.
B.2.2 From orthogonality to hidden units to arbitrary functions: an infinite width perspective
A critical point is that Theorem B.1 requires the estimation error to be orthogonal to any measurable function g(x) of the input. However, the EBD algorithm, as practically implemented, enforces orthogonality of the output error (e = - fz( to the activations of the network's 's hidden units, h
(k) (z). The question whether satisfying these more limited orthogonality conditions sufficient
approach the true MMSE estimator. We argue that in the limit of infinite network width, this can indeed be the case.
The argument relies the universal approximation capabilities of wide neural networks:
19

---

Rahimi and Recht showed that a hidden layer with random i.id. Gaussian weights and appropriately chosen biases (e.g., uniform for Fourier features) can linearly approximate functions within a corresponding Reproducing Kernel Hilbert Space (RKHS) Ha associated with a shift-invariant kernel k [51, 52]. Specifically, for a function in Hes N such hidden units (random features) can achieve an expected (xx) approximation error of order 1/VN, assuming the input distribution Px has compact support.
Building on this, Sun et.al. analyzed random ReLU networks, where hidden units are of the form ReLU(w" x) with w ~ (0, I) [53]. They demonstrated that the RKHS induced by the corresponding random feature kernel is dense in L2 (2(xx) under mild conditions on Px. This establishes the universality of random ReLU features for approximating any square-integrable function with the linear combination of that hidden layer units.
Our initial setting for the EBD Saamework-specfficce a first hidden layer with ReLU activations initialized with random i.i.d. Gaussian weights is essentially the same the random ReLU feature setting analyzed in [53]. Although weights change during training, one might hypothesize that in the infinite width limit and with sufficiently controlled learning rates (to limit the deviation of weights from their initial distribution), the set of first hidden layer activations ()g (1(2) (x)) could retain its universal approximation capability. This is an area for future rigorous analysis.
Under this crucial assumption that the (potentially trained) first hidden layer activations {h can form basis that is dense in Lz Px) N(1) oc, an estimator that achieves perfect orthogonality with these activations.
(1) (x)) w f=1
state the following result concerning
Theorem B.2 (Convergence to MMSE for Estimators Orthogonal to a Dense Basis of First-Layer Activations). Let fe(x) be estimator for y given Assume its error eg(x) y fe(x) satisfies the orthogonality condition with respect to set of N() first-layer hidden unit activations (h) (x)
(ot) 00 fori= =1 .....,, N()..
(22)
Further assume that the linear span of these activations (h." (x)) is dense in L2(Px) as (11) + co. That is, for any glx) E L2((x) and any e O, there exists a sufficiently large ( and linear combination g =mE c,hy (x)) such that:
NO.
(o(z))g(0(p)) (x) (Pxx)) Se.
(23)
Then, as + 0 (corresponding to the infinite width limit where N(1) oc), fe( converges to the optimal MMSE estimator fMMSE(œ) in the Lz(Px) sense:
[[fMMSE(=) - fe(x))l<a(PX) +
Proof. Consider the inner product between the error of the estimator fe(x), ce(x) = - fe(x),
and arbitrary function g(x) E L2(Px). We can write:
yy - fe(x),g(x)) = (y - fe(x), - (x) + gz))
+ (v (
(()))))()())))
The second term, (v - (e(l)).g())a
-(v- fo-) Σ, og (x)(()) Σn cf
fe(x), (x)). is zero due to the assumed orthogonality condition (22). Thus, are left with the first term. Applying the Schwarz inequality and using the denseness assumption Eq. (23):
Ky fe(x),
Ky fo(x). -(0)
< yy -
- ()) (x))Le((Px)
S (z) Ec
(24)
This shows that as E -> 0, the emoor (œ(x) becomes orthogonal to any g(x) E Lz(Px), i.e.
(y - (())))(((())) + 0.
20

---

Now, let fMMSE(() be the tree MMSE estimator. By definition, its enror eMMSE(*) fMrsS(()) orthogonal to any g(x) L2(xx), (y- fam O. We can rewrite the left side of Eq. (24) as:
v - [e()) = ((y [MMSE(1)) + (uums((t)-ff(x)).(()))
00 -
(fMMMSE(*) fe(z).
Substituting this back into the inequality Eq. (24):
KfMMSEE(>) - fel))) ((=))) [[e(())l(a(Px)*
(25)
This inequality holds for any g(r) E (2xx)). We can choose g(x) = fMMSS(()) fo(x) (assuming this difference is in L2 (. This yields:
[lfMMME(*) - fe Ta( ((())) S [cco())lla(P)
As + 0 (corresponding to N(1) + oc under the denseness assumption), and assuming the error of the estimator (z) [[e(())l remains bounded, the right-hand side approaches zero. Therefore:
[[fMMSE())- - f((x))l +
which implies fe(x) + (MMSE(I) the L2(Px) sense.
This theorem is significant because it elucidates the properties of an estimator that achieves the specific orthogonality targeted by the EBD algorithm with respect to its first-layer hidden units. Theorem B. established that orthogonality to all measurable functions is sufficient for MMSE optimality. Theorem B.2 demonstrates that, estimator's 's error is perfectly orthogonal to its first-layer hidden unit activations, and if these activations form a dense basis (a condition motivated by infinite- width random feature networks), then such an estimator indeed converges to the true MMSE solution. This provides a theoretical rationale for the EBD algorithm's 's objective: by striving to decorrelate output errors with hidden unit activations, EBD aims satisfy the premise of this theorem. The result offers theoretical justification for how achieving orthogonality with respect to a practical, finite set of internal network features can, under idealized conditions of network width and feature richness, lead to overall MMSE optimality. The "meaningfulness" of these orthogonality constraints, even in high-dimensional spaces, is thus linked to the expressive power of the network's learned features. The theorem hinges on the denseness of the feature space generated by the first hidden layer and the perfect satisfaction of the orthogonality conditions. Formalizing the conditions under which EBD effectively approximates these conditions and the extent to which universality of features is preserved during training remain important directions for future research.
Discussion of Theorem B.2:
B.2.3 EBD framework objective
The EBD framework designs loss functions that aim to satisfy the orthogonality conditions discussed. Specifically, for network [x),, the target is to achieve:
(y fe(x), 11 (x)) 0, for all output components, layers k, and units j.
(26)
By minimizing decorrelation losses based on Eq. (26), particularly for a universal first layer as analyzed in Theorem B.2, the network is guided towards the MMSE optimal solution. The practical EBD algorithm uses empirical estimates of these correlations and updates network weights to minimize their magnitudes.
B.2.4 Scaling of orthogonality conditions for nonlinear MMSE
The generality of the stochastic orthogonality condition in Eq. (18) for nonlinear estimators allows, in principle, for an even greater expansion of the number of constraints. If the error e, (z) of the optimal MMSE estimator is orthogonal to any gz)), it is also orthogonal to any function of its own hidden (k) unit activations, 9m (h (x)), since h
(k) (x) is itself function of Thus, one could enforce:
(m (CEnD,(()) mm (h, (k) (x))) = E [ernn.(x) pm (h) (k) (x)) 0.
(27)
21

---

for = P. 1,.... and for a set of M different nonlinear functions (.v(.). This extension could potentially introduce greater diversity of updates and more strongly enforce the conditions for MMSE optimality. While this approach theoretically increases the number of constraints and might offer benefits, it also increases computational complexity and has not been pursued in our current numerical experiments. The practical benefit versus the added cost of enforcing orthogonality against more complex functions of hidden units remains an open question. The primary EBD algorithm focuses on 9m (h) (K)) h
C The derivation of update terms
In this section, present the detailed derivations for the EBD algorithm and its variations, as introduced in Section 2.3.
C.1
A and Ab, calculation
In Section 2.3, we defined the weight update elemet [AW1] f1yy as follows:
aG (k) [m]"
ÇTr
R (k) [m
[m
aw:4k (k)
The derivative term in this expression can be expanded as
(n oe (hi MB + 21) (M)) mmB + 21)h (k-1)
(mB + (1) [mB + 11)h (k-
JmB
T
(k))) ml @w(kk 8/
[mB +
e;
g/( (hg (1())) (m ( 1)k [m + 1)B))n ( [m + 1)
where e; represents the standard basis vector with all elements set zero, except for the element at index i, which is equal to I.
By defining the matrix
(²) [m]
R ge
[(m]
q (k) (m +
(0m 1)
which represents the projection of the output error onto layer k we can express the weight update as:
(m+1)B
A W (8) [m]li= c
Σ
(m]) (M)) (u (k) (m)))) (x (m]n5t->)(p))
n=m++1
To further simplify this expression, introduce the matrices:
G [m]] = g (1) (k) (h(((((())) lm B+B+ + 1)) g (k) (k) [mB [m
2])
g( (1() (m1BB)) ]. (28)
F (k) [m]
(m p(a)) (u - 1]) ((R) (k) [mB + 2 f(t)( (u()) + B) ]. (29)
and (k)) l (k)
[m]
Fll [m]] @ @(k) [m]. which allows us to express the weight update in
more compact form:
A AW (k) [m] < (x) (m H [m]]"
Following a similar procedure, the bias update is given by:
Abi [m] cz (k) [m] 1x
22

---

C.2 2 and Ab, calculation
In Section 2.3, we defined the weight update element A2l1 2/15 involving the derivative of the output еттог as
aE m aw
ÇTr
R ge
G ( [m[]]]
(k)
1j
To begin, we consider the derivative term:
de
(k)
which can be expanded as
de
de
(h(L)
Ou(4)
Oh(L-1)
aw
=
Oh L.)
Ou(L)
0h(1-11
@u(4-1)
ding (1) (1(22))
w())
ding( (wll 113)
Oh (k+1)
@ulk+1)
(k)
Ou(k+1)
Oh(k)
Tu(k)
aw(R)
diag( (u((t)))))
w(++1)
(ia (u(t)))
19
This expression reflects propagation terms, from the output back the layer Defining ((() diag(f (LL) (m(2)(n)) and
4 [|]
o (x+11) 1) [n] ((++1) [m]] mmldiag(f') (1)) (p((
(u(*) (m]))
we obtain
dc(n]]
[m]h,
Injer
aw (k)
Thus, the derivative of the error at time step with respect to w:(h) can be written as:
CTr
R (k) ge
[m]
aE
m
G
(k) (mm)
mm
ow
(M)
(m++)) 1)B
dein]
ÇTr [m](m)) [m]
Σ n=mB+
(m))
(8)
(n
aw
Substituting the definition
g
(k)
[n]
Rgg [m]] [all'g(( (1 (1(*
(k)
we obtain:
CTr
Reghhon im
R
θE G [m)]
aw (k)
(m+1)B
CTr
Σ
h3
(k 1)
[n]
(K) (njeg (m)
n=
(m+1)B
=
Σ
hh (k- 1h
(k) [n] 2f & (b)) (nfer.
(k)
n=mB+1
(m+1)8
T
s (k) In Tg |m]h (k- 1)ー ej.
e
n=mB+1
Now, defining:
( [nj
[rs].
23

---

and assembling these into the matrix:
(b) ml
wi (m + 1]
(m + 2]
[mm +)B] 1
we can compactly express the weight and bias updates as:
Aw (k) [m] (w)( [m] H [m]²
Ab2 ml (
(k) m Bx11
C.3
Update corresponding to the layer entropy regularization
In Section 2.4.1, we introduced the layer entropy objective as
1
E hh (K)
ml]
log det (R), [m]] + c(0
(30)
2
where,
R (k) m Ag
(1) h
mm 1 (1 Ag)
H
(k)
[m] H (k) [m]".
(
and,
m = h
m B + +1
(+)
[Im + B)
(31)
The derivative of the entropy objective in Eq. (30) with respect to Wij is given by
aJ (k)
E [m]
(k) a h [m] + (
Tr [ (m(t
aw(k) (R)
aw?h)
2(1 Xx) Tr
(R
=0) H [m]
(k [m]] aw (k)
this expression, the derivative term can be explicitly written as
(H (k)
m)
W
((-
[m] (2))
F (k)
m
aw (k)
aw,
F d
m
(e.e) T H (k [m]]
c, di.
m
1) mm]
aJ (k) [m]
2(1 >xE)
Tr (R (k) [m] + 1
- [m] (F (k)
[m] @ H (k- [mm)) e?")
di,
(w:(k) (R)
B
2(1
>xE) TT
(((R) (k) [m] + (()) [m] [m]) (k (mm]] [m]] e,,e?"
2(1 >A.) 0/ ((((1, (k) [m] + 1)
(k [m|= e,
2(1 Xσ)
[( (m) + ( (m]) [m] 1) pm [7]
Consequently, we obtain
Waa (k) [m]
2(1 - >E) (k) [m] I) - (m() @ F (k) [m]) [m]]H( (m11)m]] (32)
(((R),
B
Through a similar derivation, we also obtain
boa (1) [1) 2(1 >α)
(K) [m] + - (((R))
- [ml) @ F, (3) (2) [m]]1nxI.
(33)
24

---

C.4
Update corresponding to the power normalization regularization
In Section 2.4.1, we introduced power normalization objective as,
2
(m++1) 1)B
(m())
Σ
Σ
b [n]
-pp)
B
N=m++1
The derivative of this objective with respect to wh(h)
can be written as
aJP
(2)
[m]
4
(m+1)B
Σ h
(2)
[n]
1
(m+1)B
Σ
h
(k)
[n)
(ul)
on aw!!
W (k)
B
(k)
n=mB+1
B+
di(mn)]
(m+1)B
[m]
[n] p(k) (())((()))) (n)))(
72
Therefore, the gradient of the power-normalization objective with respect (()) can be written as
Vwi 1/9 [m]] = D[m](H')) [m] F (
4
[m]
(34)
B
where [[m] diag(di (di [mr]], [2[mm],.... .... d [m]). The gradient with respect to ((k) can be obtained in similar way as
bo (1) m
4 + D[m] [m] F 1xx1
(35)
B
C.5 On the EBD with forward projections
In the EBD algorithm introduced in Section 2.3 output errors are broadcast to individual layers to modify their weights, thereby reducing the correlation between hidden layer activations and output errors. To enhance this mechanism, we introduce forward broadcasting, where hidden layer activations are projected onto the output layer. This projection facilitates the optimization of the decorrelation loss by adjusting the parameters of the final layer more effectively.
The purpose of forward broadcasting is enhance the network's ability minimize the decorrelation loss by directly influencing the final layer's 's weights using the activations from the hidden layers. By projecting the hidden layer activations forward onto the output layer, we establish direct pathway for these activations to impact the adjustments of the final layer's 's weights. This mechanism allows the final layer to update its parameters in way that reduces the correlation between the output errors and the hidden layer activations. Consequently, the errors at the output layer are steered toward being orthogonal the hidden layer activations.
This mechanism could potentially be effective because the final layer is responsible for mapping the network's internal representations the output space. By incorporating information from earlier layers, we enable the final layer to align its parameters more closely with the features that are most relevant for reducing the overall error.
While the proposed forward broadcasting mechanism is primarily motivated by performance optimiza- tion, it can conceptually be related to the long- rangrange [54] and bottom-up [55] synaptic connections in the brain, which allow certain neurons to influence distant targets. These long-range bottom-up connections actively being researched, and incorporating similar mechanisms into computational models could enhance their alignment with biological neural processes. By integrating mechanisms that mirror these neural pathways, forward broadcasting may be useful for modeling how information is transmitted across different neural circuits.
C.5.1 Gradient derivation for the EBD with forward projections
We derive the gradients of the layer decorrelation losses with respect to the parameters of the final layer. The partial derivative of the objective function (*k (h()) E) with respect to the final layer
25

---

weights can be written as:
(. (h(*) (M) c)
R Retheo)elm] a ml (*) (m 17)
[m] = CTr
ow (2)
aw
1/
E ml aw
CTr
Reghtenellm [m]
(2) [m]
(L)
(m+1))
CTr R Re(h(o))am)
Σ
θe[nl ow
ghh() (m))")))))
R=m
Substituting the definition [m derivative as:
[n]
R Re [m]" [m)() (n), we can further express the partial
(R) 0J
lm+1)B
[m]
CTr
Σ
hj (n [n17]
(1-
@w!!
(1)
B+1
(m+1)B
<
Σ
h
1h [m] [n] [n) (nles:
n=mB+1
(m+1)
T
Σ
[n] [0 (m] ej,
e
n=mB+1
(m+1)B
$
Σ
(f
[n]) @
og (0 (L]) ej.
n=mB+1
Next, defining the following terms:
with ()th ('n(2(())) 2 [m),
and assembling them into the matrix:
Vp (k. [m]] = ()( [mB + 1] 26((k,1)) [mB +
v((m + 11BB ],
we can write the weight update as:
AW(LA)) [m] = (k.bk ( (() [m]
Following a similar procedure, the bias update can be written as:
Ab (b..))).(mm (m(k-0mmIBxx
Based on these expressions, we can write
1n+
A (L,))) [m]les
Σ
p (1) [n]]]/ (n)))))))))) In]h
n=mB+1
(m++1)B
(L,)) [m]]: (m]]) 5 Σ
f(T'' (u
(L)
[n]]a: (k) [n].
26

---

D Additional extensions of EBD
D.1 Extensions Convolutional Neural Networks (CNNs)
Let H ( E ER ((P) xM(xx xx())
represent the output of the k layer of Convolutional Neural Network (CNN), where p(k) is the number of channels and the layer's output is M x N dimensional.
Furthermore, we use W(kp)) IR ppppt) x xfx*) and b (kpp) R to represent the filter tensor
weights and bias coefficient respectively for the channel-p of the k layer, and (2 is the symmetric convolution kernel size. Then convolutional layer be defined as
H (k.p) = ((((k))))
(36)
u(hm)) (k-1) W(k.)) + ((Pp))
(37)
where the symbol "*" represents the convolution operation that acts upon both the spatial and channel dimensions to generate the pth channel of hthh layer output H (k.)), and f is the nonlinearity acted on the convolution output.
D.1.1 Error Broadcast and Decorrelation formulation
Similar to Eq. (3), we have the cross-correlation between output errors and the arbitrary function of the hth layer activation of the pt channel denoted as g(*) (H (H)),, for each layer and spatial indexes E Z < S M (8) and E: : 1.11 < (R)
R (H( lg,
Elg
(k) H (k:p) [r, s)),,] = 0.
(38)
Then this cross- cors-coorelation must ideally be zero due to the stochastic orthogonality condition. We can then write the loss for layer-k batch-m as:
H (kp)
{m
1
R (y q,
(39)
2
where R. (H is the recurrently estimated cross-correlation using the training batches. Then can optimize the network by taking the derivative of the loss function with respect to the weight (k.p) corresponding to input channel h and weight spatial indexes i, Z i, < (K) as hij
(k)()) H P.P) cm
ow (k.p)
hij
ne (m+1)B
RR (Roun Im, q .12 0g (H (H.)))) :l)
Σ
1nlr] Tr
ow
(k.p)
(40)
-=1 mB+1
hij
n1
1)
Og (k) (k2) [, :)
Σ
Σ
R [m|m]
(RameHinn)eel (m, 4, ④
aw
(k.p)
1 n=mB+1 r,S
haj
[r.s]
in which n. is the error dimension, (T)) and (R) the width and height of the htth layer, and the derivative with respect to the c term is neglected. The inner partial derivative term can be written as
0g (k)
HH (k.p) |n.::]
(k) (H [R, ;
(41)
dw
(k.p) hij
(k.p) ow
hij
and using the Eq. (36) and Eq. (37).
0H (p
(n.
(kk() 2 (((k) H (n,
(42)
@w (k.p) (k.p)
hig
Although we call it as convolution, in CNNs, the actual operation used is the correlation operation where the kemel is unflipped.
27

---

where ((6) hig E R
is a Kronecker delta tensor that occurs as the gradient of W(kp)
with respect to whin hij
Combining the expressions, we have
Re Eea [n]] (Reo)HH 4,4 O g (8) (H (n, (n,;,:) (m, (43)
p(n,p.;d 1.:]
Then, combining the Equations (40), (41), (42), and then writing the convolution explicitly, we have
(J(h) (H (e)p))))ml
(m+1))B
= Σ
Σ
pl..p...
(E (kk11) :1) (na)]
awle))
(R.p)
lig
n=m++1 8,8
(m+11)B
=
Σ
Σ
pln,p,r,8)
Σ
hay
aI(K-1M0 [n, +
R=mB+1 7.5
W
By the definition of the delta function
((h)) Ehyy and writing the resulting expression as a 2D convolution
between H{k-1) and @ respectively, have
(m+1))B
Σ
Eomnpprrs H (k- (m, i,
== B-1 11 ris
(1)++1)
Σ
pin,p.;: * H 1..) m, 15
12 1
The resulting expression for the weight update is:
(J (H)))),)((m =5 Σ
(m-+1BB
(o1.. ((k
(44)
(w(kp)
n=mB41
Similarly, it can be shown that the bias update:
(k) (K(p)) mm]ml
(m+1)BB N(A) My())
Σ
Σ
Σ 01r,p,,8))
Ob(kp)
The convolutional layer parameters can be trained using these gradient formulas for each layer separately, and can be calculated by utilizing the batched convolution operation.
D.1.2 Weight entropy objective
The layer entropy objective is computationally cumbersome for convolutional layer that has multiple dimensions. Therefore, we propose the weight-entropy objective to avoid dimensional collapse
(k) ( (k))
1
log det det(R (R (i + (*))))
2
where we define (1) tensor (A), and the covariance matrix R Rypi is conditionally defined as:
E P(4 x 1( (4() (a())
as the unraveled version of the full size weight
W (kyT (A(A)
if p(k)
R W
(k)
(k)r'
w (k)r'
otherwise.
decrease its dimensions and reduce the computational costs for further steps. Then, the derivative of this objective can be written as:
W
if P(k) > p(k-1)),20)0..p)
W
E
R (A)
W
otherwise.
Therefore, (1) [.
can be obtained by reshaping AJ (k) (W(+)) as the weight tensor ((k)
F.
k53
28

---

D.1.3 Activation sparsity regularization
To further regularize the model, we enforce the layer activation sparsity loss that is given as
H
(.p)))
(P)))
(45)
The gradient of the sparsity loss with respect to the hidden layer can written as:
|H
H (k)))
sign(H H (b))
H
(bp))
(46)
|H(*))||
(k.))) 12
Then, the gradient of the loss with respect to the model weights can be calculated in a similar manner as the Eq. (44):
(m++))
Σ
AJ
H (k.p)
p,,
* H (g(k-1,A) (1, [n,;,]]
ow (k.p)
B
n=mB+1
D.2 Extensions to Locally Connected (LC) Networks
represent the output of the hth layer of Locally Connected Network (L.C), where P()) is the number of channels and the layer's output is (() ( dimensional. We use W(kp..,a) E Ii (2) and b (k.p.r,,) E R to represent the filter tensor weights and bias coefficient at spatial locations Z 1 S I ((k and & E & S 8 S N(k),, for channel-p of the bthh layer, where (2kk) is the local receptive field size. Then a locally connected layer can be defined as
Let (k) E R p x
HR.pp) (((((H.))), (k--)) - (H w(lk)) + (b.)))
(47)
(48)
where the symbol "@" represents the locally connected operation which acts upon both the spatial and channel dimensions, but without weight sharing across spatial locations, generating the pa channel of the 4t layer output H(bp)), and f is the nonlinearity applied to the result.
D.2.1 Error Broadcast and Decorrelation formulation
For the LC network, the stochastic orthogonality condition and the corresponding loss (H)())))) cm(mm for layer-k at batch-m can be written equivalently as Eq. (38) and Eq. (39) respec- tively. Then the optimization can be performed by taking the derivative of the loss function with re- (k...s.))
corresponding to input channel h, weight spatial indexes i,j E Z S i, s (())
spect to hij
()( H (k.p) ()))))m()) aw (kpp.)))) hij
(m+1)B
cg/1] Tr (R (Re q, - (k) (k) (m,
Σ
(k.mi))))))
(49)
q= nom m B+1
hij
(m+ 1)B
0g (k) (H (k))) (n D
2 Σ
Σ
( (m ; sy|n)
aw (k....))) hij
q=1 n=mB+1 r.s
[r.s]
The inner partial derivative term can be written as
0g
(k...)))) (w(...)))) (H(k)) (n,;;:) =g g ((() (H()))) (n,;;)) o
HH (k.p)) [n, aw (k.p.))))
(50)
aw
hij
hij
and using Eq. (47) and Eq. (48), obtain:
0H (k.)) (k
(((((s @ (E(
((k-))(n,... [m,
(51)
(k...)))))
hij
29

---

c(k.r,,) Here, hij E x0l00x0 x M(k), x ((K) is a Kronecker delta tensor that occurs as the gradient of w(kk) with respect to W (p.ps))) hij
Combining the expressions in Eq. (49), Eq. (50), Eq. (51), and the expression for e as in Eq. (43) which is equivalent for both CNNs and LCs, and then writing the locally connected operation explicitly, we have
a (H())), (m
(m++1)
Σ
Σ
pmn,p;;: o (glk.))))
(Eni @ H (n, (n,,;,)))
::]]) [raa]
aw (.pr))) lig
n=m
(m++))
=(
Σ
p,
Σ
elkiri
W W, - H(k (n, + +57
hij
n=mB+1
Then, by the definition of the Kronecker delta, the resulting expression for the weight update is:
aw (k.p.)s)) (M (H(M)), c)lml
(m+1)B
=
Σ
(oln.p.r.s)
iis
(52)
hij
Similarly, it can be shown that the bias update is:
(.((k) (H(Kp) {mmm
(m+1)B
Σ n=mB+1
$ln.p,r,8)
(b(k.r))))
D.2.2
Weight entropy objective
Similar to CNNs, propose the weight-entropy objective avoid dimensional collapse in LCs
5 (w()))
1
log det wii + s gl)) I),
where we define (k) E P x 1 M (1 (20 (k the unraveled version of the full size
weight tensor (kk,, then the covariance matrix R wiy defined as:
R Rw = w (k:J (R)
Then, the derivative of this objective can be written as:
AJ (W(k)) = W
w (A)
aw (ummn) can be obtained by reshaping A (k) (W(+) as the weight tensor W(A). 6i
8Jg (K))
D.2.3 Activation sparsity regularization
The layer activation sparsity loss for the LC is the same as the one given for the CNN in Eq. (45), with its gradient with respect to the activations as in Eq. (46). Then, the gradient of the loss with respect the model weights can be calculated in a similar manner as the expression Eq. (52):
(h (K)
[m)
(m+1)B
Σ
J H Jn,p.r @ (gk-1,b) [n r + 8, + +
Ow OW(.. (k.....)) hig
B B+1
30

---

E Gradient alignment in EBD
E.1 Alignment between EBD updates and backpropagation gradients
To investigate the relationship between the EBD update directions and the gradients produced by backpropagation (BP), we analyze the cosine similarity between the EBD update vectors and the corresponding BP gradients throughout training. This analysis quantifies how well the EBD learning dynamics align with traditional gradient-base based optimization methods.
We conduct experiments on two architectures: a 3-layer multilayer perceptron (MLP) and a locally connected (LC) network, both trained CIFAR-10. Figure and Figure illustrate the cosine similarity between EBD updates and BP gradients at each training epoch.
Training Test
Hiddee
Hidden Layer
Bodt Accurncy
0.5
0.5
2000000
0.4
0.4
0.3
0.4
8.0
0.0
50
20300
40000
50000
20000
40000
00000
00010
Figure 3: Cosine similarity between EBD updates and backpropagation gradients in 3-layer MILP trained on CIFAR-10. Alignment is consistently positive and improves during training.
Layer
Layer
3.20
Couiti Siriality
2.0
Thaining
20033 NNES) JOKES MC20 SOIEES 6C386
36020 260330 JXIEO 00052 SIC3O AOCDO)
Layer2
Hidden Layer
Acorvey
04-
120
125
200
Codine Smiaahy
Cos 110 Hanntt
03
30030 200090 SONOCC #0000 500ETO 4C2000
300300 300000 30000 00059 300000 900990
Figure 4: Cosine similarity between EBD updates and backpropagation gradients in locally connected network on CIFAR-10. Positive alignment indicates directional consistency between EBD and BP.
These results demonstrate that EBD update directions are not arbitrary but align with the descent direction of the loss function indicated by BP, supporting its effectiveness gradient-free freee but principled optimization strategy.
31

---

E.2 On gradient truncation and biological plausibility
The decorrelation objective used in EBD naturally decomposes into two sets of parameter updates layer k:
(AW1
(8) ) and (AW? )
Here, (AW(") Ab( corresponds to updates that modify the hidden representation to reduce correlation with the output error, while (AW(A) )b corresponds updates that aims reshape the eтrоr signal itself.
For reasons of local learning and biological plausibility, EBD retains only the (Aw Ab(")) component and drops the crror-happing terms (AW2)) A thereby avoiding the backward propagation of gradients through the network.
To assess the impact of this truncation, we measured the cosine similarity between the full gradient (which includes both components) and the truncated update used in EBD. As shown in Figure 5, the truncated update direction remains consistently aligned with the full gradient throughout training on CIFAR-10 10 using a 3-layer MLP. This positive alignment suggests that the retained component is sufficient for effective learning, validating our simplification.
Training and Test Accuracy
Hidden (%639.99 positive)
Hidden 19100 positive)
Taining
1.0
1.0
Accuracy
0.5
Cosine Similariyy
Simmiaaity
AcC0/00)
0.6
0.7
0.5
0.5
0.2
0.3
0.2
50000
20000
50000
80003
40000
60000
80000
Ecration
Figure 5: Cosine similarity between the full decorrelation gradient (including (AW(k) Ab2 the truncated EBD update (only (Aw: (8) Ab(())). Positive similarity confirms that the truncated update remains valid descent direction.
(k) and
32

---

Background on online Correlative Information Maximization (CorInfoMaxx based biologically plausible neural networks
Bozkurt et.al. recently proposed a framework, which we refer as CorInfoMax-EP to address weight symmetry problem corresponding backpropagation algorithm Bozkurt al. [45]. In this section, we provide a brief summary of this framework.
The CorInfoMax-EP framework utilizes online optimization setting to maximize correlative information between two consequitive layers:
Σ 10 [m]] [ly [m] h00 [m]]l
where 7(+) (h()), h ((++)) [m] is the correlative mutual information between layers and + 1, and
the term on the right corresponds to the mean square error between the network output h h(*) [m] and the training label y/m]. This framework utilizes two alternative but equivalent forms for the correlative mutual information
f(ew) (R) h )
1
log det (R [m] ExT) log det R_a+t [m] + EE.)),
1 f((w) (R), h ) [m] log (R [m] + + log det R (R [m] + cxT).
2
2
c.
defined terms the correlation matrices of layer activations, i.e., Rg and the correlation matrices
of forward and backward prediction errors (R_(4+1)) and R_,())) between two consequitive layers. Here, forward/backward prediction errors are defined by
[n] h (k+ [m] w( [m]]((*)
c, [m] h (x) w(b (m]]h [k++))()
respectively. Here, W( [m]] (W((.)) (m]) the forward (backward) prediction matrix for layer
In order to enable online implementation, the exponentially weighted correlation matrices for hidden layer activations and prediction errors are defined as follows:
X
m
R (4)) [m]]
m
[m]
(1) [m]]
Am
1 A
(k)
(k) T
R Rigi [m]
[m]
[m]]
Am
x - Am
(k)
- (k)
T
R. (4) [m]] =
[m]
[m]
Through the trace approximation of log det)) function, we obtain:
log det
R (a+1) mm] + =1]
(++1) w(() Sf.. [m] ( ex W (k) ff. [m]]] + Ne+1 log(ex)
Ek
log det
R [m + +xxl)
(k)
w (k)
fb.. [m] [m]h///hhH (++0 + ex ((()) f6, [m]l] N₂ log(Ex),
Ek
F.1
The derivation of the CorInfoMax network
Based on the definitions above, the following layerwise objectives can be defined:
Jg (h [m)
f((a-1) (h (k-1) (m)) + plEx) (h h (8++))[m) for k = 1.... L 1,
33

---

ie.. correlative information maximization objectives for the hidden layers, and the mixture of correlation maximization and MSE objectives for the final layer
J (h [m] (h [m)]
h (4) [m] [ y(m]]l
The gradient of the hidden layer objective functions with respect to the corresponding layer activations can be written as:
1 c (M)
1 -- (k)
hi (h( 2n B [m/]h [m]
[m]
[m].
(53)
Ek
Ek
where y _ and B B [m] = (R 00 [m]] + ck 1) ie., the inverse of the layer correlation
matrix.
For the output layer, we can write the gradient as
V Vite JL = T (
((b) [m]
((h mm]
yyml).
CL-1
The gradient ascent updates corresponding these expressions can be organized obtain CorInfo- Max network dynamics:
du (8) sm;s) -uuu (8) [m;
1 M (l)
(k) [m;
(k) s].
[m]]
[m]] h [m;;
EK-1 Ca
s]
e
[m;
ds
5k
Ek
(m; s) = (*() (m; s)
w 1) [c] (k- (m; s):
/,
(k) [m; (*) (8) [m; s]
w (k)
[m
mllh (k+1) [m; s].
fb
h [m;; = a (u [m;; s])
where m the sample index, 8 the time index for the network dynamics, Tu is the update time con- stant, M [t] = Eu (2y [t] + gu= and 04 = min( (1, max(u, 0)) represents the elementwise clipped-Rell ReLU function, which is the projection operation corresponding to the combination of the nonnegativity constraint (*) and the boundedness constraint (A) || S 1 on the activations of the network.
Note that Bozkurt et al. [45] takes one more step to organize the network dynamics into a form that fits into the form of network with three compartment (soma, basal dendrite and appical dendrite compartments) neuron model.
F.2
CorInfoMax-EP learning dynamics
The CorInfoMax-EP framework in Bozkurt al. [45] employs equilibrium propagation (EP) to update feedfoorward and feedback weights of the CorInfoMax network.
E.2.1
Feedforward and feedback weights
In the CorInfoMax objective, feedforward and feedback weights correspond to forward and backward predictors corresponding to the regularized least squares objectives
(k) [m]) en Crr(W(
W/ (k) [m]ll
+ lle
(k+1
2:
and
Cro(W [m]) = e w [m] l|
(k) [m]]]
respectively. The derivatives of these functions with respect to forward and backward synaptic weights can be written
aC5s ff [m])
= 2c wilh [m] - 2c (k+1)
(1)
aw (K) ff [m]
34

---

and
y0 (W (ml])
(8) 2 W ( [m] 2e (k)
[mm]h(+1mmm [m]²
oW(k) fb [m]
The EP based updates of the feedforward and feedback weights are obtained by evaluating these gradients in two different phases: the nudge phase (3
3 > 0), and the free phase (3 0):
/w//
(e
(R+1)
[m]h
(m)]
3=3 (C
(R+1)
[m (b) (m²²")
g¹
3=0
[m] B
(e
+ (k)
(m)
(0(())
(
(k)
(k] [m]
[m]
3=8
F.2.2 Lateral weights
The lateral weight updates derived from the weight correlation matrices of the layer activations, using the matrix inversion lemma [43]:
((k) I 1] (B m] (2( (m)) where 2 (m) (B [m] B=
F3 CorInfoMax-EP
Although the CorlnfoMax-El EP algorithm derivation above is based on single input sample based updates, it be extendable to batch updates. Assuming a batch size of B, and define the following matrices:
H (b) [m] = h [m + 1]
(m + 2
(m0 1)B) ].
as the activation matrix for the layer-k,
E mm]
(k)
=
+
(R) mm +
((k)
(mB + 2
(k) [Im + 1)B) ]
(54)
as the backward prediction matrix for the layer-k,
E [m]
(k)
=
(R)) [mB +
(k)
(mB 2
-(k)
(m 1) ].
(55)
as the forward prediction matrix the layer-k:
z 100 [m] = z (k) [mB +1]
(k)) [mB +
[m 1)B] |
(56)
as the lateral weights' output matrix for the layer-k, and
E c(mB + 1] c/mB + 2]
<[(m + 1)B] },
as the output error matrix.
In terms of these definitions, Algorithm 2 lays out the details of the CorInfoMax-EP alggootmmm:
35

---

Algorithm 2 CorInfoMax Equilibrium Propagation (CorInfoMax-EP) Update for Layer A
Require:
Learning rate parameters λg, (((Fk)) [m], ( [m]
Require:
Previous synaptic weights W I (forward), W(h [m (backward), (A)
(lateral)
Require: Batch size
Require:
Layer activations H ( [m]] preactivations U [ma]. output errors E [vr]. lateral weight
(A) outputs [m)) forward prediction errors E [m]] and backward prediction errors E <A())
[m]
computed by CorInfoMax network dynamics described in Bozkurt al. [45]
Ensure:
Updated weights [m], W(b.)) [m]] _B (8(((m)
1: JE <
1-Ag Am
Update forward weights for layer
k:
2(a [m)]
(k+1)
(b+1)
2: Aw( EP
[m]] f
E
mm
[ml]]
(E
[m] (k) [m] [m]"))
[m
B8
8=0
6=0
W(
[m] W(. 1 AWW
(
[m]
Update backward weights for layer k:
(1d6,1) [m]]
(k)
(k)
4: Aw(k
(b.k) [m]
E
[m]
H
[ 7)
(E
[m)
(m]])
[m]7]
B3
$=
e=0
5: W((.)) mm] +- W(..)) im 1 AW
(b.k)
[m]
EP
Update Lateral weights for layer k:
6: AB [m] < YE (+) [m
(k) [m]
B
7: ( [m] f 1 ((k) [m] + AB [m]
Ax
G
Implementation complexity of the EBD approach
In this section, analyze the computational and memory complexity trade-offs of the proposed Error Broadcast and Decorrelation (EBD) approach.
G.1
Complexity analysis: Error Propagation vs. Error Broadcast
Considering the standard MLP implementation outlined in Section 2, compare the memory and computational requirements the error backpropagation and error broadcast approaches as follows:
G.1.1 Delivering output error information to layers
Memory Requirements: For the standard backpropagation algorithm, the error is propagated through the transposed forward filters ( Consequently, no additional memory is required for the weights used in error propagation. In contrast, the error broadcast approach uses error projection matrices (k) which require memory storage of O (NN(() (y))) for each layer. Thus, the total additional storage requirement for broadcast weights given by O(EL) N())
Computational Requirements: In the standard backpropagation algorithm, propagating the error to layer & (from layer i + 1) requires O (B)(()) ((++1)) multiplications per batch, where the batch size. On the other hand, the broadcast algorithm projects the output error to layer k, requiring O(B (AB (²)) multiplications. Additionally, the projection matrix (k) is updated at the end of each batch using the Hebbian rule. Therefore, the overall computational complexity for the EBD approach is:
O(B + ()(1))
When the number of output elements (L)) is significantly smaller than the hidden dimensions (k), the computational cost of the broadcast algorithm is lower. Furthermore, the error projection
36

---

operations can be implemented in parallel for all layers, whereas backpropagation must be executed sequentially.
G.1.2 Additional cost of Entropy Regularization
Memory requirements: As described in Section 2.4.1, layer entropies are based on the covariance
matrix R (8) which requires ((v())))) additional memory storage. In a computationally optimized
implementation, storing the inverse covariance matrix B, (k) may be preferred, though still requires the same memory allocation.
Computational requirements: The main computational load is due to the computation of the gradient of the layer entropy function. The expression for the gradient obtained in Appendix C.3 as
V Vwa J(h) m]
2 1-Ax
(R) (4) [m] + +e H @
f (((()) H [[]--[]]= b (00))1))))))) 17] (H (mm)",
(2) [m]. We can divide the where for each batch, we update the layer correlation matrix matrix R computational requirement into following pieces:
Correlation Matrix Recursion Recall we have
R (k) [m] Ag R, (b) mm 1 +
1 - AE H [m] (* [m])"
In order to form R (8) [m]] explicitly at iteration m, we must compute H (m) (H (x) [m])* and
then add the result to Ag (x()(m [m - -1].
The matrix-matrix product (A)) [mm] E R times its transpose in R BBx(0 yields an (() x (A) matrix, costing (
Adding A R, (k) [m 1] that product is another ((N()))) operation, though usually smaller in comparison to the product above if moderate.
Hence the complexity of forming the new correlation matrix R (k) [m]] at each iteration is
((N()2 B).
Naive Matrix Inversion and Gradient Computation
(ax) (a) Once R [m] is formed, we need to invert R [m]] + e evaluate (1 [m]] and its gradient. Naive inversion of an (1) x (K) matrix is ((N(())3)) After this inversion, we multiply (k)
(R [m] - - by ( [m] E ( xB which costs ((N((k))2 B). Next, we do the ele- mentwise multiplication (0) with f'(W [m] + (T)) costing O(N(k) B). Finally, we multiply by (Hl) T Bxxx((-) which costs ( (k)-))).
Summing these terms, the dominant operations in naive update and inversion are:
1. Forming R, (k) [m]]: (NV()2 B).
2. Inverting (R),
(k) [m] + e I): )(
3. Multiplying inverse by H (²)) [m]: O ((N(K))) B).
4. Final multiplication by (H [m]) (NN()) B N()-
Thus, overall cost per batch is
o (N (N(1))s3 (N())) + N(O) B
which could be simplified to
((((H)) + (N(0))2 + (O) B B(-
If (k) is large, the cubic term (N)3 associated with the matrix inversion typically dominates.
37

---

An alternative is to update the inverse of the correlation matrix incrementally using the fact that
XE (k) [ B
R [m] = dE R I 1| +
so the new correlation matrix differs from the previous one by a low-rank term of rank at most min(N(k), B). Neglecting the 6 term, the recursion for the inverse can be obtained using matrix inversion lemma [43] as
B h
[m] = X5
B (8) [m 1] V[m]
A I+ ( [m] mm]"[[m))
VimiT)
(57)
1- 1->B
where Vim] B Im (*) [m).
The corresponding computational cost is:
Two matrix-matrix multiplications of shape (N x N(A) with (N(k) x B), costing
(( B).
Inverting the B B matrix, costing O(B²) if B is not too large.
Thus, the update of the inverse alone costs
((N(L))) B B²).
instead of ((N())))) If < (2(k, this can be a large savings compared to the naive cubic cost. Once this updated inverse is in hand, the subsequent multiplications to form the gradient (e.g. (R), (k) [m]] + etc.) still take (((N(())2 B ((( ) B N((-1)). Overall, for each new time step m, the dominant costs become
((NVM))) B B3 + (N(A) + N(k)) B B((--)))
which is usually simplified to
B + N BN(K-1)).
Hence, using the Woodbury identity is beneficial whenever B is much smaller than N because ( B + B & (N()))33
G.1.3
Additional cost of Power-normalization
Memory requirements:
The power normalization described in Section 2.4.1, involves power
estimate parameter per hidden unit, so will require additional (T(k) storage elements for the layer-k.
Computational requirements: The gradient for the power normalization regularization function derived in Appendix C.4 takes the form:
V (8) [m] = D[m](H'') (m) [m] F( (k())
4
Based on this expression, the required number operations per batch for layer-k is N(E-I)
Therefore, we can consider the impact of the power-normalization on memory and computational requirements as negligible.
In Section 1.7, provide empirical runtime results for the EBD algorithm, relative to the backpropa- gation algorithm. These experimental results show 7 to time increase in the runtimes of the MLP model with the EBD algorithm (employing entropy regularization) relative to the BP algorithm. The runtime increase is less for CNN and LC models.
Finally, we note that the implementation complexity analysis provided above is for the MLP based EBD approach. For the biologically more realistic CorInfoMax- networks, entropy maximization is implemented through lateral weights (see Appendices F, F.2.2 and H), whose update requires O ((N()))2 multiplications per sample.
38

---

H
On the biologically plausible nature of Entropy and Power-normalization updates
As discussed in Section 2.4.1, the layer-entropy and swwer-nommalization objectives are introduced to avert potential collapse of network coefficients in the EBD algorithm. A natural question arises re- garding the biological plausibility of the EBD framework when these regularizations are incorporated. We address this question by examining specific cases:
MLP Implementation with Entropy and Power-normalization regularizations (Sec-
tion 2)
2. CorInfoMax-EBI implementation (Section 3.2)
H.1 MLP implementation with Entropy and Power-normalization regularizations
In Section 2, we presented an MLP-based EBD framework that uses batch-SGD to optimize the feedforward weights with EBD, along with entropy and power-normalization losses. As outlined in Sections 2.3 and 3.1, the gradient-based updates of the EBD loss naturally reduce to three- factor update rule, which is considered biologically plausible. We examine whether adding the layer-entropy objective in Eq. (11) and the power-normalization objective in Eq. (9) preserves this biological realism.
H.1.1 Power normalization- SGD updates
Appendix C.4 derives the gradient expression for the power-normalization loss:
W P [m]
D mm] [m] @ F
(m))
B
Focusing on an individual element of this matrix gives
mB+B
Wo (k) [m]lij
B d [m]
Σ
1 (K)) (u, (nl) nnt [m].
(58)
n=mB+1
This update depends only the activations of the neurons connected by the synapse Way. thus satisfying a local learning rule. However, the summation over the batch index in Eq. (58) might be considered biologically implausible unless the batch size B 1. In practice, one interpret the summation for B > 1 as an integral of local updates over the time window corresponding to the batch, which may still be reasonably viewed local integration in biological setting.
H.1.2 Layer entropy regularization-basee SGD updates
Appendix C.3 derives the gradient of the layer-entropy objective:
V wan ((k) [m] 2(1 >g)
(k) [(RR
[m] e
(
@ Fa (2)
H [m]"". (59)
B
h
Examining an individual element of this gradient shows
Ww (m]li (1))
2(1 >E)
Σ B=mB+1
mB+B
0 [l] (m (uu [()
where (x) (R, in the layer, the layer-entropy update for feedforward weights is not strictly local and hence violates the criteria for strict biological plausibility.
(2) [m] + ( h (A) [n]. Because (*) [n] depends on all neurons' activations
Nevertheless, this limitation is circumvented by the CorInfo EBD approach, wherein the lateral (recurrent) weights, rather than the feedforward weights, implement the layer-entropy maximization objective. We discuss this in the next section.
H.2 CorInfoMax-EBB implementation
Section 3.2 introduces more biologically realistic network by combining the CorInfoMax woanwork-knnowe to yield recurrent networks closely reflecting biological dynamics-wil with the proposed EBD approach to enable a three-factor update rule in supervised learning.
39

---

H.2.1
Power-normalization based SGD updates
In CorInfoMax-EBD. we adopt the same power-normalizttonn gradient in Eq. (58) for updating feedforward weights. Therefore, by setting the batch size B 1, these updates remain local and thus biologically plausible.
H.2.2 Layer entropy maximization
As summarized in Appendix CorInfoMax networks inherently include layer-entropy maximization via the correlative-information objective. Crucially, this entropy maximization is implemented through laterul weights the RNN structure rather than by modifying feedforward weights. Specifically, from the gradient of the correative-information objective (see Eq. (53)):
1
hoo Ju (h (2) [mm] 2yy B [m]] [m]]
[m]]
m
(60)
6
ER
the first term, 2yy s [m] (*) [m], corresponds to the layer-entropy maximization. Here, the lateral weight matrix 0 [m]] approximates the inverse of the layer correlation matrix. As described in Appendix F.2.2 (and in [45]), the lateral weights can be updated by an anti-Hebbian rule:
B p 1] X, (B(k
22 (m)2 [m]] [m)²),
where
z (8) [m] B (8) [m]] (*()) [ml]]] B=gr
Once again, this update is strictly local if 1 = 1. while for B 1, the rank-B extension may break strict locality. We demonstrate Section that CorInfoMaxxEBD with B = 1 yields comparable or superior performance to CorInfoMax-El with larger batch sizes.
H.3 Summary and conclusions
In summary, the CorInfoMax-EBD implementation described Section 3.2 offers a more biologically plausible approach to supervised learning compared to the MLP-based EBD approach in Section due to several factors:
Using lateral weights to impose layer-entropy maximization in biologically realistic manner;
Employing feedforward/feedbacck weights for forward and backward predictive coding;
Adopting neuron models with distinct compartments (soma, basal dendrites, and apical dendrites);
Incorporating EBD updates, which naturally embody a three-factor learning rule; and
Leveraging power-normalization updates, which satisfy local-learning constraints when 1
These features stem from the CorInfoMax-ER framework [45], enhanced by proposed EBD-based regularizations. This architecture reconciles the benefits of layer-entropy and power-normalization objectives with the demands of biological plausibility.
40

---

I Supplementary on numerical experiments
The models were trained on an NVIDIA Tesla V100 GPU, using the hyperparameters detailed in the sections below. Each experiment was conducted five times under identical settings, and the reported results reflect the average performance. We used the standard train/test splits for the datasets, with MNIST comprising 60,000 training examples and CIFAR-10 1-10 comprising 50,000, while both datasets included 10,000 test examples. The MNIST dataset [56] is made available under the Creative Commons AttrbbutionShhan University of Toronto, publicly available for academic research purposes. Both datasets were accessed via standard deep learning library functionalities.
Alike 3.0 license. The CIFAR-10 10 dataset [45], originating from the
Rather than utilizing automatic differentiation tools, we manually implemented the gradient calculations for the EBD algorithm, utilizing batched operations to ensure computational efficiency. As a side note, the (1 - X) factors present the derived update expressions are absorbed into the learning rate constants and thus eliminated. In our experiments, we trained the MLP models for 120 epochs and the CNN and LC models for 100 epochs on MNIST and 200 epochs on CIFAR-10. In addition, we trained the CNN model for the CIFAR-100 dataset for 300 epochs, and the CorInfoMax-EBD (3-layer, batch size 20) model for epochs.
I.1 Architectures
The architectural details of MLP, CNN and LC networks for the MNIST and CIFAR-10 datasets are shown in Tables 4 and 5, respectively, while the CNN model used the CIFAR-100 experiments is detailed in Table 7. The structure of the MNIST and CIFAR- 10 models are the same as in the reference [9], while the CIFAR- 100 model closely matches [6]. differing only the MaxPool shape. In all architectures, we used ReLU as the nonlinear functions except the last layer. Furthermore, the architectural details of the biologically more realistic CorInfoMax network for MNIST and CIFAR-10 datasets are shown in Table 6. These techniques are the same as examples in Appendix J.5 of [45].
Table MNIST architectures. FC: fully connected; Conv:
convolutional; LC: locally connected. FC layers are reported by hidden size. Conv/LC layers are reported as (channels, kernel size, stride, padding). Pooling layers use stride report the kernel size.
MLP
Convolutional
Locally connected
FC1
1024
Conv1
64, 3 x 3, 1,
LC1
32, 3 x 3, 1,
FC2
512
AvgPool
2 x
AvgPool
2 x 2
Conv2
32, 3 x 3, 1,
LC2
32, 3 x 3, 1, 2 2
AvgPool
2
AvgPool FCI
FC1
1024
1024
Table 5: CIFAR-10 architectures. Conventions are the same as in Table
MLP
Convolutional
Locally connected
FC1
1024
Conv1
128, 5 x 5, 1,2
LC1
64, x 5, 1,2
FC2
512
AvgPool
x
AvgPool
2
FC3
512
Conv2
64, 5 x 5, 1,2
LC2
32, 5 x 5, 1,2
AvgPool
2
AvgPool LC3
x
Conv3
64, 2 x 2, 2,
32, 2 x 2, 0,
FC1
1024
FC1
512
Table 6: CorInfoMax architectures. Conventions are the same as in Table 4.
MNIST
CIFAR-10
FC1
500
FC1
1000
FC2
500
FC2
500
41

---

Table 7: CIFAR-100 architectures. Conventions are the same as in Table
Convolutional
Conv
96, 5 x 5, 1,2
MaxPool
2
Conv2
128, 5 x 5, 1,2
MaxPool
2
Conv3
256, 5 x 5, 1,2
MaxPool
2
Dropout
FC1
2048
Dropout
FC2
2048
Softmax 100
1.2 CorInfoMax-EBD
In this section, offer additional details regarding the numerical experiments conducted with the CorInfoMax Error Broadcast and Decorrelation (CorInfoMax-EBD) algorithm. Appendix 1.2.1 elaborates the general implementation details. Appendix 1.2.2 presents the fundamental learning steps of the algorithm, which are based on the EBD method. Appendices 1.2.3 and 1.2.4 discuss the initialization of the algorithm's variables and describe the hyperparameters. Finally, Appendix 1.2.5 ( 3-Layer and batch size=20, 3-Layer batch size= and Appendix I.2.6 (10-Layer batch size= I) detail the specific hyperparameter configurations used in our numerical experiments for the MNIST and CIFAR-IO 10 datasets. In Appendix I. 10 we present the accuracy and loss learning curves for the CorInfoMax-EBD shown in Figures 6g(g)-() and Figures (h( respectively.
1.2.1 Implementation details
We implemented the CorInfoMax-EBBi algorithm based on the repository available at GitHub 2, This repository from Bozkurt al. [45], used basis for our CorinfoMax-EBD implementation, did not specify an explicit license in its public repository at the time of access. Our use and modification are for academic research purposes, building upon the published scientific work presented [45]. The following modifications were made to the original code:
We simplified the algorithm by reducing it to a single phase. Specifically, removed the nudge phase, during which the label is coupled to the network dynamics. In this modified version, the network operates solely in the free phase, where the label is decoupled from the network. This change aligns with the removal of time-contrastive updates from the CorInfoMax-Ei algorithm.
Reduction a single phase:
Algorithmic updates: We incorporated the updates outlined in Algorithm
Hyperparameters: We maintained the same hyperparameters for the neural dynamics in the original code. Additionally, new hyperparameters specific to the learning dynamics were introduced, which are detailed in Appendix I.2.4.
In the CorInfoMax-EBD implementation the following loss and regularization functions are used
EBD loss: (kk,
Power normalization loss: 1/
l weight regularization (weight decay): j(h()
Activation sparsity regularization: J(z (H) lh
42

---

1.22 Algorithm
The CorinfoMax-EBD algorithm follows the same neural dynamics framework detailed in [45] for computing neuron activations. Consequently, we only outline the steps specific to the learning process, which distinguishes it from the original CorInfoMax-EP algorithm described in [45]. The full iterative process for updating weights the CorInfoMax- EBD algorithm is provided Algorithm 3.
Algorithm 3 CorInfoMax Error Broadcast and Decorrelation (CorInfoMax-EBi) Update for Layer
Require: Learning rate parameters λg.Ag, ((( [m], ((F-)0m)) (0(h2) [m]
Require:
Previous synaptic weights W( (m- 1] (forward), W (b.k) [m 1] (backward), ( [m
1] (lateral)
Require: Previous error projection weights R (g(h0)) -1]
Require: Batch size B
Require: Layer activations H [m]] in Eq. (13), the derivative of activations ((k) d in Eq. (29), in
Eq. (5), prediction errors E and E (R)
in Eq. (54)-Eq. (55), lateral weight outputs ( in Eq. (56)
computed by CorInfoMax network dynamics described in Bozkurt et al. [45] (and Appendix F)
Require: The nonlinear function layer activations G in Eq. (4) and the derivative of the nonlinear function of layer activations G(A) in Eq. (28)
Ensure: Updated weights W(S)) (m]. W(b.k) [m]. [m))m)
Error projection weight update for layer k:
Xd
1: Rge [m] + Aa Rg [m 1]
G (R)) Em]
B
Project errors to layer k: Rgge (k)
2:
Q(k) [m]
[m]]
E [m]]
Find the gradient of the nonlinear function of activations for layer k:
3: a
[m]
F mm]
[m]
( [m]
Update forward weights for layer k:
4: (W(.k)
(f.k)
[m]
(x(d,,8)
ml
(R) mm (b-))
EBD
5: A (f.))
Pred [m]
((.
m
(R)
[m]]
(H (mr))
B
6: W(S) [m]
(f mm
1] +
(f.k)
[m]
(f.k) Pred [m]
EBD
Update backward weights for layer k: (db.))) [m]]
7: A (b.k)))
EBD [m] <-
4e
(R) [m] [m]]
B
8:
(b.b) (bbk
Pred [m]]
(b,A)
m
(R)
[m]
(m++) [m]]
B
9: W(..)) [m] W(b.k) [m 1] Awwhh
EBD [m] + A
(b.k)
AW(h (h,8) Pred [m]
Update Lateral weights for layer k:
EBD [m] <- (1) [m]]
AB(R) (R)
[m] (
B
11: AB [m] <- 7E
(8)
ZE
(k)
[m]]
(k) [m]]
((k) [m] xg
(B(k) + AB
(A)
[m]] + AB (A) ( [m]
43

---

L2.3 Initialization of algorithm variables
We initialize the variables W(rk), W(b,), and o using PyTorch's Xavier uniform initialization with its default parameters for the MNIST dataset. For the CIFAR- dataset is initialized with gain 0.25. For the lateral weights (*),, we first generate a random matrix ((k) of the same dimensions, also using the Xavier uniform distribution, with gain= 1 for the MNIST dataset and with gain= 0.5 for the CIFAR-10 dataset. then compute (k)) [0] = J (4) (k),, definite symmetric matrix.
ensuring that B (0] a positive
1.2.4 Description of hyperparameters
Table 8 presents a description of the hyperparameters used in the CorInfoMax-EBI implementation.
Table 8: Detailed explanation of hyperparameter notations for the CorInfoMax-EBI algorithm
Hyperparameter Description
a/m] og [m] p
Learning rate dynamic scaling factor
Learning rate dynamic scaling factor
Leurning rate decorrelation loss (forward weights)
Leurning rate decorrelation loss (buckward weights) Learning decorrelation loss (lateral weights) Learning forward prediction
Learning backward prediction
Leurning power normalization loss
Target power level
Leurning rate activation sparsity (forward weights)
Learning rate for activation sparsity (backward weights)
P
Forward weight éz-regularization coefficent
Backward weight Errregularization coefficent
Layer correlation matrix update forgetting factor
A
Emor activation cross correlatio forgetting factor
m())
Monentum factor for decorrelation forward weight gradient
Batch size
1,2.5 Hyperparameters for 3-Layer MNIST and CIFAR-10 Models
Table 9 and 10 summarizes the hyperparameters used the 3-layer CorInfoMax -BD experiments for the MNIST and CIFAR-IO 10 datasets with batch size of 20 and respectively. The iteration index denoted by m in all expressions.
Table 9: 3-Layer CorInfoMax-EBi hyperparameters for MNIST and CIFAR-10 datasets (B 20).
Hyperpurameter
MNIST
CIFAR-10
a/m]
3 x
3 x [
anfmm
3x L
3 [+] +1
[96] 60, 165]e/mm]
[80, 50, 1e5]cr/m) for epoch= 0
[320. 400, 1c5 ] for epoch> 0
a ml]
[06, 60, 1e5]a//m]
[o, 0, 0ia/m]
alaa ml
00.25, 0.25, 0,25]a/m/ for epoch=
[0.5, 0.5, afm for epoch=
(0.5, 0.5, 0.5]a[m) for epoch> 0
2.0. 2.0, 2.0 for epoch>
u mn
[0.11 x 10
0,06 x 10 18, 0.035 10 38]a(m]
[0.11 x 10 0.06 10 18, 0.035 x 10 18)afm]
n (i. mm)
[1.125 x 10 10-16, 0.375 x 10 10-15]a(m]
[1.125 10 10-1550.375 0.375 x 10 [0-25]a/m)
ana ml
111 10 10-3, 6,666.. 10
[14 x 10-3, (6x:00-2 10001))
[2.5, 2.5, 0.1]
[255, 2.5, 0.1.
arg [m]]
(0.008, 0.135, 0Jog[m]
[0.008. 0.135, 0jog[m]
Par [m]]
0.35. 0.05]o2im"
[o, 0.35. 0.05)o2(m)
8
8 10
10-2
10-2 m
(4)
8 10
x 10-2
10-2 I +1+1
10-2 I
Am
0. 0.999999
0.999999
Aa
0.99999
0.99999
0.99
0.999
0.99
0.999
m
m0
B
20
20
44

---

Table 10: 3-Layer CorInfoMax-EBD hyperparameters for MNIST and CIFAR-10 datasets (B
Hyperparameter MNIST
CIFAR-10
afm]
3x 10 18
3 10-3 LS0.
arfin]
3 L [4.8, 3.0, 10
3 [x] +1
ale [mmo
2.5, 10"]a(m] for epochh
[16, 20, 5 10 for epoch>>
ata mm]
[18, 3.0, 5 x 10m]
[0, 0ja(m]
ullai ml
0..0125, 0.0125, 0m for epoch= 00005,, 0.025/as/m) for epocht>
[0.025. 0.025, 0,025/a/m) for epoch=
0011, 0.1, 0.1'a(m] for epoch>
00.11 10-18, 0.06 10-18, 0.035 10
00.11 10 0.06 10-18 0.035 10-1*)a(m]]
ael mm)
[1.125 18, 0.375 10 15/m[m]
[1.125 10 16, 0.375
15|a[m]
(os ml
[2. 10 3 10-1, 3.5 10 [0a[m]
[2.2 x 10 3.x 10-", 3.5 10-""]oa[m)
225,, 2.5, 0.1]
(0.125, 0.125, 0.005]
nee [mn]
[0.0004, 0.00675, 0f
[0.0004, 0.000675, Djag[m]
Pa. [m]
[0, 0.0175, (m0
[o, 0.0175, 0.0025)
(/ [m]
8 10
8
10-2 x 10 +1
10-2 HT +1
Po Imll
(8
10 +1
10 I +1+1 +1
999999990
0,99999930
Aa
0.99999
0.99999
0.99
0.999
1
0.99
0.999
1
+
+o
B
1.2.6 Hyperparameters for 10-layer CorInfoMaxxERB MNIST and CIFAR-10 datasets for batch size
Table 11 list the hyperparameters used in the 10-Layer CorInfoMax-EBD numerical experiments for the MNIST and CiFAR-10 10 datasets with batch size of 1. In these experiments, weight thresholding scheme is applied to the network weights for every 5000 samples, where the weights with 0.00003 scale (relative the peak) are set zero.
Table 11: 10-Layer CorInfoMax-EBD hyperparameters: MNIST and CIFAR-10 10 datasets (B = 1).
Hyperparameter
Value
a/m]
3x +II+T
oa[m]
3xT
uddr [m]]
3.5
3.5
3.5 6e4 Ja(m)
n da
(m
0.03afm] 11x10
cra
[m
0.11001,0110111111111
0.1 0.1.0.1..1111111111101 0.035 1e(-18)-ofm)
m
1,1,04, 1.000,0044000 U.4. 0.4 0.4, 0.4444444
(1e(-18) a(m)
[m
2,,2,,,,,2,2..1 2,5,
1e(-3) a2|m]
1, 1,, 1,2,
re [m]]
0.16, 0.16, 0.16, U.16, 0.16,0.166 0.16,0016. 0.16, 0.16, 0.0 [o2[m]
[m]
01110
Mi.-e, [m]]]
a000mm111111 11
Ag
0.99999095
Ad
0.9999907 (1 700.99900000 with
midd(()) [m]]
0.99
0.999(1
B
1
45

---

I.3 Multi-Layer Perceptron
In this section, we provide additional details about the numerical experiments conducted to train Multi-layer Perceptrons (MLPs) using the EBD algorithm (MLP-EBD). Appendix 1.3.1 outlines the implementation details of these experiments, while Appendix 1.3.2 discusses the initialization of algorithm variables. Information about hyperparameters and their values for the MNIST and CIFAR-10 datasets can be found in Appendices 1.3.3-13.4. In Appendix I.10 we present the accuracy and loss learning curves for the MLP architecture, shown in Figures 6.(a)-(b) and Figures (.(a)-bb) respectively.
I.3.1 Implementation details
For the MLP experiments using the proposed EBD approach, we adopted the same network architec- ture described in [9], detailed in Tables 4 and 5.
In the MLP-EBD implementation, the following loss and regularization functions were employed:
EBD loss: ((k),
Power normalization loss: J/P
Entropy objective: J
l weight regularization (weight decay): J(h).
Activation sparsity regularization: J
(
Additionally, we imposed a weight-sparsitg constraint by setting WS percent of the weights to zero during the initialization phase and maintaining these weights at zero throughout training.
I.3.2
Initialization of algorithm variables
We use the Pytorch framework's 's Xavier uniform initialization with gain value 10-2 on the Ry variables, and Kaiming uniform distribution with gain 0.75 for synaptic weights W())
I.3.3 Description of hyperparameters
Table 12 provides the description of the hyperparameters for the MIP-EBD implementation.
Table 12: Description of the hyperparameter notations for MLP-EBD.
Hyperparameter
Description
a[m] az(m]
Learning rate dynamic scaling factor
Learning rate dynamic scaling factor
uld.b.))
Learning rate for (backward projection) decorrelation loss Learning rate for (forward projection) decorrelation loss Learning rate for entropy objective Learning rate for power normalization loss Target power level
pla
(K)
Learning rate for activation sparsity
(h)
pw-ls
Weight fz-regularization coefficent
Ag
Layer autocorrelation matrix update forgetting factor Error-layer activation cross- coos-correlltion forgetting factor
Xa
(d))
Momentum factor for decorrelation gradient
B
Batch size
WS
Weight Sparsity
46

---

L33,4 Hyperparameters for MLP-EBD on MNIST and CIFAR-10 Datasets
Table 13 summarizes the hyperparameters used in the MLP-EBD experiments for the MNIST and CIFAR-10 datasets. The iteration index is denoted by m in all expressions.
Table 13: MLP-EBD hyperparameters for MNIST and CIFAR-10 datasets.
Hyperparameler
MNIST
CIFAR-IO
[mm]
1,5 x Lio.
10_
+1
1,5 X 10. +1 T6_ 3 x 104
0g ag]m]
+1
+1
3 10a
a [mm]
18000 a amjmmgggg ml for k 0,
(4000, 2000, 2000, 3500 [[m])2[()
20000 a 02 m for k
u 1
0.005a(mmaa2(m)) for k = 0,1
0.005a[mla2(m/ for 0,1
[m]]
[2.5 10 1.5
a
2.5 10 1.5 10 1.5 10 0]
u ulo (m
x 10-3, 6 x 10-3. x 10-10 a[m]
4 10-3, x 10-3, 6 10-3, x 10 a(m] 0.25, 0.25, 0.25, 0.1]
[0.25, 0.25, 0.1]
(k)
[0.8, 0.3, 0J []]]
[08,, 0.3, 0.3, 0J a)m)
th
Ha-tz
1.6 x 10=*a[m] for all layers
1.6 x 10-*afmm for all layers
As
0.99999
0.99999
da
0.999999
0.999999
mld)
0.9999 all layers
0.9999 layers
20
20
WS
55
40
47

---

1.4 Convolutional Neural Network
In this section, we offer additional details regarding the numerical experiments for training Convolu- tional Neural Neural Networks (CNNs) using EBD algorithm (CNN-EBD). Section I.4.1 provides information about implemetation details. Appendices I.4.2 and 1.4.3 discuss the initialization of the algorithm's 's variables and describe the hyperparameters. Finally, Appendix 1.4,4 detail the specific hyperparameter configurations used our numerical experiments for the MNIST, CIFAR- and CIFAR-100 datasets. In Appendix 1.10 we present the accuracy and loss learning curves for the CNN, shown in Figures 6.(c)-(d) and Figures 7.(c)-(d), respectively.
1.4.1 Implementation details
The architectures we utilized for the CNN networks can be found in tables 4 and 5 respectively for the MNIST and CIFARIO 10 datasets. the training, we used the Adam optimizer with hyperparameters 31 = 0.9, 32 0.999, and 10 [57]. Also, the model biases are not utilized. In the CNN-EBD
implementation following loss and regularization functions as detailed in section D. are used:
EBD loss: J(k),
Entropy objective: (k)
Activation sparsity regularization: J
Specifically for the CIFAR-100 experiments, we applied both training and test time augmentation of data to improve model generalization and handle increased difficulty in the task. At training time, each image was randomly translated into 2-pixels with reflection padding horizontal flip that ensures that every image is flipped every other epoch, reducing redundancy compared to standard random flipping. During evaluation, we used test-time augmentation combining horizontal flipping and multi-crop averaging six translated views (the original, two one-pixel translations, and their mirrored counterparts).
deterministic alternating
1.4.2 Initialization of algorithm variables
We use the Kaiming normal initialization for the weights, with common standard deviation scaling parameter ow, on both the linear and convolutional layers. Furthermore, the estimated cross- correlation variable R, (linear layers) and R (H(t)) (convolutional layers) are initialized with zero mean normal distributions with standard deviations dR... and OR.on. respectively.
L4.3 Description of hy peprpaarseetess
Table 14 describes the notation for the hyperparameters used to train CNNs using the Error Broadcast and Decorrelation (EBD) approach.
48

---

Table 14: Description of the hyperparameter notations for CNN-EBD.
Hyperparameter
Description
dexp ali
Exponential learning rate decay parameter. Learning rate dynamic scaling factor where is the epoch index Learning rate for (backward projection) decorrelation loss Learning rate for entropy objective
a ulddb))
((E
Pe
(k)
Learning rate for activation sparsity
ow
Standard deviation of the weight initialization.
Std. dev. of (1()) initialization in linear layers
Rcoo
Std. dev. of R (H) initialization in convolutional layers
Rioeal
Gain parameter for ( initialization locally connected layers
X
Layer autocorrelation matrix update forgetting factor
Ad
Error-layer activation coss-coorelltion forgetting factor
An
Convergence parameter for À as in Equations (61), (62) Entropy objective epsilon parameter for linear layers Entropy objective epsilon parameter for conv. or locally con. layers
B
Adam Optimizer weight decay parameter Dropout probability
p
B
Batch size
We also introduce a convergence parameter AR which increases the estimation parameter for the decorrelation loss λς together with the estimation parameter for the layer entropy objective Äg, to converge to as the training proceeds with the following Equations (61), (612) where is the epoch index:
x(+1)= A + An (1 (1- (0) i 0.
(61)
, 2 O. (++1) = 2 + ARR (1- (6) g
(62)
L4.4 Hyperparameters for MNIST, CIFAR-10 and CIFAR-100 datasets
Table 15, lists the hyperparameters as defined in Table 14, used in the CNN-EBD training experiments.
Table 15: Hyperparameters for CNN-EBD for the MNIST, CIFAR-10 10 and CIFAR-100 datasets, where denotes the epoch index.
Hyperparameter
MNIST
CIFAR-10
CIFAR-100
Ocep
0.97
0.97
0.97
ole
10-4. 10-4.ogo
Oosp
10 4
aes
10-4. 4.00p
0.1afi) for
0,1,2,3
0.1a for = 0,1,2,3
0.1afd for k 0,1,2,3,4
10a for
10ofi fork
10a for
[1
10
100
1100 a
5 [10-7a(f]
PM
[1
1
10
0
10
Pafi]
1 10a 0 10
of
0 0 710
VT
O
1e-2
Ie
Ic 0.99999
0.99999 Ic 0.99999
Aa
0.99999
N
0.99999
0. 0.99999
1e
le
1e
1e
le
le
B
P
N/A
N/A
0.075
49

---

1,5 Locally Connected Network
In this section, offer additional details regarding the numerical experiments for the training of Locally Connected Networks (LCs) using EBD algorithm (LC-EBD). Appendix 1.5.1 provides information about implemetation details. Appendices I.5.2 and 1.5.3 discuss the initialization of the algorithm's 's variables and describe the hyperparameters. Finally, Appendix 1.5,4 detail the specific hyperparameter configurations used in our numerical experiments for the MNIST and CIFAR-10 10 datasets. In Appendix 1.10 we present the accuracy and loss learning curves for the LCs, shown in Figures (.(e)-(f) and Figures (f())-(f), respectively.
1,5.1 Implementation details
The training procedure mirrors the CNN approach described in Section 1.4.1 for CNNs. In the LC-EBD implementation, the loss and regularization functions detailed section D.2 are used:
EBD loss: J(k),
Entropy objective: (()
Activation sparsity regularization: J
1.5.2 Initialization of algorithm variables
We use the Kaiming uniform initialization for the weights, with a common standard deviation scaling parameter ow, on both the linear and locally connected layers. The estimated cross-correlattin variable ()) (linear layers) is initialized with a normal distribution with zero mean and standard deviation R.... Also, the parameter R (g((()()))))) )e (locally connected layers) is initialized with Pytorch framework's Xavier uniform initialization with the gain parameter equal Riocal
1.5.3 Description of hyperparameters
Table 14 in the CNN section, again describes the notation for the hyperparameters used to train LCs using the Error Broadeast and Decorrelation (EBD) approach. The convergence parameter ÅR introduced in equations Eq. (61) and Eq. (61) is used as well.
1,5.4 Hyperparameters for MNIST and CIFAR-10 datasets
Table 16, lists the hyperparameters as defined in Table 14, used in the LC-EBD training experiments.
Table 16: Hyperparameters for LC-EBD for both the MNIST and CIFAR- 10 datasets, where denotes the epoch index.
Hyperparameter
MNIST
CIFAR-10
Ccep
0.96
0.98
alil
10 Gep
10 0esp
(
0.1a for k = 0,1,
0.5a for k = 0,1,2,3
10ali for k =
5a/i for k 4
10a
0 10 [((]
1
10
10 110 a
(
1 1
10
|10-1aaf1
[ 1
10
0 ]
100118
ow
RR...
Recal
1
le 0.99999
dd
0.99999
Ag
0. 0.99999
0.99999
AR
3e
3e
B
1e
eL
1e
1e
1e
le 5
B
16
16
N/A
N/A
50

---

1.6 Implementation details for Direct Feedback Alignment (DFA) and backpropagation training
This section presents further details on the numerical experiments comparing Direct Feedback Alignment (DFA) and Backpropagation (BP) methods, conducted under the same training conditions and number of epochs as those used for our proposed EBD algorithm. The results of these experiments are provided Table We also include the DFA+E method, which extends DFA by incorporating correlative entropy regularization similar to the EBD. Note that, when the update on the R is fixed to initialization, the EBD algorithm reduces to standard DFA.
For BP-based models trained on MNIST, CIFAR-10 and CIFAR-100, we used the Adam optimizer with hyperparameters B1 0.9, 82 0.999, and 10 [57]. For DFA and DFA+E models, again used the Adam optimizer for CNN and LC models, while MLP models were trained using SGD with momentum.
In Tables 17 and 18, we detail the hyperparameters for models trained with BP, DFA, and DFA+E update rules MNIST and CIFAR-10 respectively. In Table 19, we give the hyperparameters for the CNN model (detailed in Table 7) trained with with BP and DFA. Some of the learning rate and the learning rate decay values methodologies are linked the tables corresponding to the hyperparameter details of its EBD counterpart, where the same method is also utilized for its DFA DFA+E counterpart. Unlinked values denote constant value applied each layer, or the step decay multiplier applied per epoch. Additionally, sparsity inducing losses are not utilized for BP, DFA and D models.
Table 17: Hyperparameter details for models trained on the MNIST dataset, including learning rate, L2 regularization coefficient, learning rate decay, and number of epochs for MLP, CNN, and LC models using BP, DFA, and DFA+E methods.
Model
Method
Leamingg Rate (1(d...)) L2 Reg. Coef.
LR Decay (Reep)
Epochs
BP
5e
1e Table 13 Table 13
0.96
120
MLP
DFA
Table-13
Table- 13
120
DFA+E
Table-13
Table-13 13
120
BP
5c
1e
0.97
CNN
DFA
Table-15
le
0.97
DFA+E
Table-15
le
0.97
BP
5e -
1e
0.96
100
L.C
DFA
Table-16
le le 8
0.96
100
D
Table-16
0,96
100
Table 18: Hyperparameter details for models trained on the CIFAR-10 dataset, including learning rate, L2 regularization coefficient, learning rate decay, and number of epochs for MLP, CNN, and LC models using BP, DFA, and DFA+E methods.
Model
Method
Leaming Rate (1 (d..))))
L2 Reg. Coef.
LR Decay (Aesp)
Epochs
BP
5e Table-13 Table-13
1e-5
0.85
120
MLP
DFA
Table-13
120
DFA+E
Table-13
120
BP
5e -
le
0.92
200
CNN
DFA
Table-15
le
0.97
DFA+E
Table-15
le
0.97
BP
1e-4 4
le 6
0.90
200
LC
DFA
Table-16
le 6
0.96
200
DFA+E
Table-16
le 6
0.96
200
Table 19: Hyperparameter details for the CNN model trained on the CIFAR-100 dataset, including learning rate, 1.2 regularization coefficient, learning rate decay, dropout probability and number of epochs using BP and DFA methods.
Model
Method
Leamingg Rate (1
L.2 Reg. Coef. LR Decay (Atss)
Dropout (p)
Epochs
BP
5e
Ie
0.97
0.5
200
CNN
00.1 0.1 0.1 1 50 0.25a/i)
0.95
0.075
300
51

---

pursue this optimization, as 1.7 Runtime comparisons for the update rules
In this section, we present the relative average runtimes from the simulations, normalized to BP for the MNIST and CIFAR-10 models in Tables 20 and 21 respectively, for the models that we implemented and demonstrated their performance Table 1.
The results show that entropy regularization in both EBD and DFA+E more than doubles the average runtime. However, these runtimes could be significantly improved by optimizing the implementation of the entropy gradient terms, specifically by avoiding repeated matrix inverse calculations. A more efficient approach would involve directly updating the inverses of the correlation matrices instead of recalculating both the matrices and their inverses at each step. This strategy aligns with the network structure. Nonetheless, chose CorlnfoMax networks already employ it effectively.
The efficiency of the DFA, DFA+E, and EBD methods can be further enhanced through low-level optimizations and improved implementations.
Table 20: Average Runtimes in MNIST (relative to BP)
Model
DFA
DFA+E
BP
EBD
MLP
3,40
7.68
1.0
8.06
CNN
1.68
2.95
1.0
3.85
LC
1.61
3.57
1.0
3.54
Table 21: Average Runtimes in CIFAR-10 (relative to BP)
Model
DFA
DFA+E
BP
EBD
MILP
2.85
6.94
1.0
7.61
CNN
2.10
3.24
1.0
4.11
LC
1.35
2.01
1.0
2.41
L8 Reproducibility
To facilitate the reproducibility of our results, we have included the following:
Detailed information on the derivation of the weight and bias updates of the Error Broadcast and Decorrelation (EBD) Algorithm for various networks in Appendix C for MLPs, D. for CNNs, D.2 for LCs,
ii Full list of hyperparameters used in the experiments in Appendix 1.2.5, 1.3.4, 1.4.4, 1.5.4, iii. Algorithm descriptions for CorInfoMax Error Broadcast and Decorrelation (CorInfoMax- EBD) Algorithm in pseudo-code format Appendix 1.2.2,
iv. Python scripts, Jupyter notebooks, and bash scripts for replicating the individual experiments and reported results are included in the supplementary zip file.
I,9 Computational resources
All experiments were conducted within a High- Meih-Peerformacce Computing (HPC) facility. Each experimental run utilized single NVIDIA Tesla V100 GPU equipped with 32GB of HBM2 memory. To provide context on execution times for our proposed CorInfoMax -BDD models:
Training the 3-Layer CorInfoMax-EBD model (as described in Appendix I.2.5) for 30 epochs required approximately 22 hours.
Training the 10-Layer CorInfoMax-EBB model (as described in Appendix I.2.6) for 100 epochs took approximately 75 hours. Execution times for other models (MLP, CNN, LC) and baseline methods were generally shorter; relative runtime comparisons are provided in Appendix 1.7.
52

---

I.10 Accuracy and loss curves
Figures and 7 present the training/test accuracy and MSE loss curves over epochs for the CIFAR-10 and MNIST datasets. Solid lines represent test curves, dashed lines denote training curves.
0.59
0.50
0.7
DFA
DFA
DTA+2
D744E
E60
EB0
0.55
100
Epochs
Lgachs
(a)
(b)
1.00
3.0
Accurac)
Accuracy
0.50
0.50
BP
BP
24
DYA
0.97
150
Epochs
(c)
(d)
1.00
3.0
0.59
2:2
Accuracy
Accuracy
0.50
0.97
0.57
40
60
50
109
25
50
75
100
115
130
375
100
Epochs
(e)
0.59
0:9
0.50
Accurocy
0.98
DFA
DFA+E
DYAEE
EBO
34
E90
0.97
125
150
375
Epochs
Epochs
(h)
Figure 6: Train and test accuracies plotted as a function algorithm epochs for various update rules (averaged over 5 runs associated with the corresponding + std envelopes) for the (a) MI. on MNIST (b) MLP on CIFAR-10 (c) CNN on MNIST (d) CNN on CIFAR-10 (e) LC on MNIST (I) LC CIFAR-10 10 (g) 10-Layer CorInfoMax-EBD with batchsize=1 MNIST (h) 10-Layer CorInfoMax- EBD with batchsize=l on CIFAR-10.
53

---

DFA D74+E
DTA+2
EB0
E9D
MSEE LOSS
MSE
Epochs
(a)
(b)
BP
DFA-E
MSE Loos
20
40
80
100
23
50
75
100 Epochs
123
150
375
200
Epochs
(c)
(d)
1e:2
MSE Loos
M5E
Epochs
Epochs
(c)
(f)
16-2
CorinfoMax.EBD (batch size=20)
CortnfoMas: E6O (batch size-20)
1.2
4.0
50
2,5
Epochs
Epochs
(g)
(h)
Figure 7: Train and test mean squared errors (MSE) plotted as a function of algorithm epochs for various update rules (averaged over n 5 runs associated with the corresponding + std envelopes) for the (a) MLP on MNIST (b) MLP on CIFAR-I0 (c) CNN on MNIST (d) CNN on CIFAR-I0 (e) LC on MNIST LC on CIFAR-IO (g) 3-Layer CorInfoMax-EBN with batchsize=20 on MNIST (h)
3 3-Layer CorInfoMax-EBI EBD with batchsize=20 on CIFAR-10.
54

---

Figure 8 shows the training/test accuracy curves over epochs in the 10-Layer CorInfoMax- MBD numerical experiments for the CIFAR- 10 and MNIST datasets. Furthermore, Figure presents the training/test accuracy curves over epochs for the CNN model trained with the CIFAR-100 dataset. Solid lines represent test curves; dashed lines denote training curves.
50
Accuracy
Layer
size-1]
L0yer
size-1)
:00
140
160
300
Epochs
Epochs
(b)
Figure 8: Train and test accuracies plotted as function of algorithm epochs (averaged over n = associated with the corresponding + std envelopes) for training with (a) 10-Layer CorInfoMax-
EBD with batchsize=l on MNIST (b) 10-Layer CorInfoMax-Eil EBD with batchsize=1 on CIFAR-10.
1.0
DFA
0.8
Accuracy
0.4
0.0
Epochs
Figure 9: Train and test accuracies plotted as a function of algorithm epochs for various update rules (averaged over n runs associated with the corresponding + std envelopes) for training with the CNN network on CIFAR-100.
55

---

J Calculation of the correlation between layer activations and output error
Figure Ic illustrates the decrease in the average abso- lute correlation between hidden activations and output error during backpropagation, using Multi-layer Per- ceptron (MLP) model with the architecture outlined in Table 5, on the CIFAR-10 dataset. Details for the MSE based training and the Cross-Entrer Eass-py based training are explained in Appendices J.1 and J.2 respectively.
Layer-1 Train
Layer-1 Test
Layer-2 Train
Layer-2 Test
0.08
Avg. Abs. Corr. Coeff
0.06
J.1 Correlation in the mean squared error (MSE) Friterion-based training
0.04
The MLP models are trained using the Stochastic Gra- dient Descent (SGD) optimizer with a small learning rate of 10 and the MSE criterion. In both plots, the initial value represents the correlation before training begins. The reduction in correlation observed during training provides insight into the principle of the EBD algorithm.
20
40
Epochs
To compute these correlations, we apply batched ver- sion of Welford's algorithm [58], which efficiently calcu- lates the Pearson correlation coefficient between hidden activations and errors in a memory-efficiene way by us- ing streaming statistics.
The evolution of the average absolute correlation between layer activa- tions and the error signal during backprop- agation training of an MLP with two hid- den layers (using the MSE criterion) on the MNIST dataset, showing the correlation decrease over epochs, on both the training and test sets.
Figure 10:
Welford's algorithm works by accumulating the neces- sary statistics (c.g., sums and sums of squares) across batches of data and finalizing the correlation computa- tion only after all data has been processed, avoiding the need store all hidden activations simultaneously.
Given the hidden activations h (k)
N(* where b is the batch size and ( is the number of hidden units, and the errors paxk, where k is the number of output dimensions (c.g.. classes); the goal to compute the Pearson correlation coefficient between activations h; for each hidden unit i and the corresponding error values across all samples
(o) E)
(k)
oo Voz +
where is small constant for numerical stability. Finally, we compute the average of the absolute values of the correlation coefficients for each hidden layer l to generate the corresponding plots.
Figure 10 shows the correlation throughout training on the MNIST dataset, employing the MLP model detailed Table 4, where we observe the correlation decay behavior similar Figure lc.
To verify that the correlation-decay phenomenon observed for fully-connected networks extends to convolutional architectures, we trained a compact five-layer CNN consisting of three U MaxPool blocks (with 32, 64, and 128 3x3 filters, respectively) followed by fully-connected layer with 512 hidden units and an output layer with 10 logits. The network was optimized for 30 epochs on CIFAR -00 using mini-bater SGD (n 10-3 momentum 0.9) and the mean-squared error criterion on one-hot labels. After every forward pass we streamed the Pearson correlation between each hidden activation vector and the output eгror using the batched Welford estimator. Figure 11 plots the epoch-wis mroch-wise evolution of the average absolute correlation coefficient for both training and test data. As with the MLP in Figure 1 all layers exhibit pronounced monotonic decline, confirming that back proppaaaaaatio progressively enforces the stochastic orthogonality of hidden activations and output errors in convolutional networks well, as expected.
56

---

conv1 Train
conv1 Test
conv2 Train
0.08
conv2
0.06
fc1 Train
fc1 Test
fc2 Train
Avg.
0.04
fc2 Test
0.02
Epochs
Figure 11: The evolution of the average absolute correlation between layer activations and the eπтor signal during back propagaopnt training of CNN on CIFAR-10. The CNN contains three convolutional layers and two connecccdde layers and is trained with the MSE criterion. The correlation decreases over epochs on both the training and test sets, mirroring the behavior observed for the MLP architecture and supporting the generality of the correlation decay across architectures.
J.2 Correlation in the cross-entropy criterion-based training
Although the stochastic orthogonality property is specifically associated with the MSE loss, also explored the dynamics of cross-correlation between layer activations and output errors when cross-entropy is used as the training criterion.
With the same experimental setup as described in Appendix J. but replacing the MSE loss with cross-entropy, we obtained the correlation evolution curves shown in Figure 12a for CIFAR- 10 and in Figure 12b for MNIST dataset. Notably, the correlation between layer activations and output errors still decreases over epochs, despite the change in the loss function.
0.07
Layer-1 Train
0.10
Layer-1 Train
Layer1
Layer-1
Layer-2 Train
Layer-2 Train
0.05
Layen-2
Layen-2
Layer-3 Train
Layer-3 Test
Avg. Abs. Corr. Coeff
Avg. Abs. Corr. Coeff
0.05
0.03
0.03
10
20
30
10
20
30
40
Epochs
Epochs
(a)
(b)
Figure 12: Evolution of the average absolute correlation between layer activations and output errors during backpropagation training of an MLP with three hidden layers, trained using cross-entropy loss. (a) CIFAR-10 10 dataset and (b) MNIST dataset. Despite the use of cross- entss-yn the correlation decreases similarly to the MSE criterion.
57

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction claim EBD is a novel, MMSE-grounded frame- work for biologically plausible credit assignment, circumventing weight transport, leading to three-factor rules, integrating with CorInfoMax, and achieving competitive better per- formance against other error-broaden methods on benchmarks. These claims are supported by the theoretical development Sections 2 and (e.g., MMSE orthogonality, three-factor learning rule derivation in 3.1) and experimental results in Section 4 (Tables I and 2). The scope, including scalability as an area for further exploration, is also appropriately stated.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 5 (Impact and Limitations" ")) explicitly discusses limitations concerning scalability, computational costs/hyperprrmmteer complexity of EBD.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
58

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The theoretical grounding of EBD the MMSE orthogonality principle is detailed in Section 2.2. Appendix A provides preliminaries nonlinear MMSE estimation. Appendix elaborates on the stochastic orthogonality condition, its sufficiency for MMSE optimality (Theorem B.1), and its application to network training, including arguments for using hidden unit activations under idealized conditions (Theorem B.2). These sections outline the assumptions and provide the theoretical arguments/proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4 outlines the experimental setup. Appendix ("Supplementary on Numerical Experiments") provides extensive details regarding network architectures (I.1), hyperparameters for all presented models (EBD variants and baselines like BP, DFA) across datasets (I.3-1.7), optimizers, learning rate schedules, and other implementation specifics necessary to understand and reproduce the experiments. We also provide all codes as supplementary to the article.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
59

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the implementation of the proposed method in the supplementary material, accompanied by README file that outlines the structure and usage instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (uttps://nips. cc/
sasiiisoicccciioioooooccooociiii oliicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //wpps.cc/publliiii http//uubl/gguddeeee httggdddeeeeeeueaaiiiiiiiaaa wnraabbissee iemiiissinni aasssnoooiicyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The datasets (MNIST. CIFAR- 10) use standard train/test splits as detailed in Appendix I (lines 1051-1053). Appendix extensively lists hyperparameters, their selection rationale or values for different models/datasets (e.g.. I.3.4-L..7,, 14.33..44, 1.5.3-1.5.4, I.6.3- 1.6.4, I.7), optimizer types (Adam, SGD with momentum), and other training configurations (epochs, batch sizes).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
60

---

7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments were conducted five times, and average and standard deviation values reported in Tables. In Appendix I, Figures 5 and 6 explicitly state that they show mean results with standard deviation envelopes from these multiple runs. Table reports mean and standard deviation for the 10-layer CorInfoMax-EBi EBD results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
All experiments utilized a single NVIDIA Tesla V100 GPU with 32GB HBM2 memory, accessed via HPC facility (details in Appendix D). Appendix 1.9 provides specific execution time examples for the more computationally intensive CorinfoMax-EBD models. Appendix 1.7 further provides relative runtime comparisons for other models and methods.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / h/w/eeuriss.ee. htte:publlcc/thhcccuuudedud
Answer: [Yes]
61

---

The research proposes novel learning algorithm and evaluates it using standard public datasets. The work is foundational. The "Impact and Limitations" section (Section 5, lines 315-318) outlines the positive aim of advancing the fields and notes that specific negative societal impacts directly from EBD mechanisms are identified beyond general MI. considerations.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 5 ("Impact and Limitations") discusses the positive societal impact by stating the aim to advance Machine Learning and Computational Neuroscience. also addresses the negative aspect stating that, a foundational algorithm, specific negative societal impacts directly attributable to EBD are identified beyond general considerations common advancements in machine learning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This research proposes a new learning algorithm (EBD) and primarily evaluates it on standard, publicly available datasets (MNIST, CIFAR-10), The paper does not introduce release new large-scale datasets or pre-trained models that would typically carry a high risk for misuse requiring specific safeguards (such those for large generative models or models trained sensitive scraped data).
Justification:
62

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The sources for the standard datasets used (MNIST [52], CIFAR-10 [45]) cited in Section and their licenses/terms of public availability (CC BY-SA 3.0 for MNIST, public availability for research for CIFAR-10)
Justification:
noted in Appendix I. The external code repository from Bozkurt et al. [45] used as a basis for the CorInfoMax-EBl is cited, and its observed licensing status (no explicit license found in the public repository at the time of access) is transparently stated in Appendix I.
implementation
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode wwt.coo////////o datastsss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The primary assets the proposed EBD algorithm and its variations, along with their software implementation. As stated the filled checklist answer for question 5, the code implementation and README file with usage instructions provided in the supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
63

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing or direct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve direct research with human subjects that would necessitate IRB approval or a discussion of participant risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMMs were not used component of the research methodology, theoretical development, or experimental analysis presented in this paper. Any use of LLMS restricted assistance with writing and editing the manuseript, which does not require declaration per NeurlPS policy.
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs as any important, original, or non-standard components.
Please refer to our LLM policy (https htte:p//eerrrpsccc/ooffeeeeeeeee cc//oonferrencce/222///111 for what should or should not be described.
64


Reviews:
[{"review_id": "LV2YiQH7ba", "reviewer": "Reviewer_sM7v", "summary": "This work develops a novel approach for solving the credit assignment in a biologically plausible way. The authors demonstrate theoretically and experimentally that using a loss function that penalizes how correlated the individual layer activations are with the output error is an effective way to train neural networks. They additionally demonstrate that their approach is an extension of a three-factor model for synaptic plasticity.", "questions": "1. Why is the performance of DFA + E not underlined in Table 1?", "limitations": "The authors discuss some of their limitations, but I think this could be discussed in more detail. The authors had to use several regularizers to prevent collapse. The authors also only evaluated on small networks and datasets. More discussion on these limitations, and others, would be helpful.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper is well written and well motivated. The key points were well summarized and the description of their method was largely clear. \n\n2. The core idea of the paper is (to my knowledge) novel and well grounded in theory. \n\n3. The numerical results demonstrate the effectiveness of the approach and illustrate its better performance than other state-of-the-art methods. \n\n4. I think this work will lead to interesting new directions and will be of broad interest to the NeurIPS community. \n\n**Weaknesses** \n\nI found no major weaknesses of this paper. There are, however, a few points I think would make this work stronger. \n\n1. The authors demonstrate that the layer correlations decay with training, when using BP (Fig. 1C). I think more on this in the main text (and not just in the Appendix J) would be good. Showing that this is true across different architectures and datasets, and comparing the time-scale over which the correlations decay vs. the time-scale over which the loss decays would be interesting. \n\n2. I found the connection with the three factor learning rule to be very interesting, but the abbreviated discussion made it a little hard to follow. In particular, it felt like the notation changed and it was not clear to me how Eq. 7 mapped onto the 3-factor learning rule. More on this would be helpful. \n\n3. In Table 1, DFA + E (for MNIST) has nearly the exact same values as EBD, but it does not have its accuracy underlined (despite cases of actual ties). Is this because the performance is slightly different and EBD wins? Or is there maybe a typo on the performance of DFA + E?  \n\n**Minor points**\n\n1. I think Fig. 1 could be made bigger, to make it easier for the reader to see what the authors are showing. \n\n2. In Eq. 3, there is the variable $R_{g \\epsilon}^{(k)}$. Then on line 140, it is denoted as  $R_{g^{(k)} (h(k)),  \\epsilon}$. Are those supposed to be the same thing or did I miss something?", "was_revised": true, "final_justification": "I continue to believe this work is of high quality and will be of interest to the NeurIPS community. For that reason, I recommend accepting it.", "rebuttal": "We would like to the reviewer for the constructive comments. \n\n### Strengths\n\n> 1. well written ... well motivated... clear.\n> 2. ... novel and well grounded in theory\n> 3. ...numerical results demonstrate the effectiveness... better performance ...\n> 4. lead to interesting new directions ... broad interest to the NeurIPS community.\n\nWe appreciate the positive comments by the reviewer.\n\n---\n\n### Weaknesses\n\n>I found no major weaknesses...few points... make ...stronger.\n>1. The authors demonstrate that the layer correlations decay with training, when using BP (Fig. 1C). ... more on this in the main text (and not just in the Appendix J) would be good. Showing that this is true across different architectures and datasets, and comparing the time-scale over which the correlations decay vs. the time-scale over which the loss decays would be interesting.\n\nWe thank the reviewer for this helpful suggestion regarding presentation. Due to space constraints, we placed the extended analysis of correlation dynamics in Appendix J, while highlighting key aspects (e.g., Figure 1C) in the main text. Following the advice by the reviewer, we included a correlation decay example  for the CNN architecture and CIFAR-10 dataset, which demonstrates that the correlation decay behavior in Figure 1.c replicates as expected. In the main text, Section 2.2 (line 125), we included a new sentence: \"Similar correlation declining trends are also observed across different datasets and architectures (see Appendix J).\" \n\nWe also agree that exploring the time scales of correlation decay versus loss reduction across architectures is an interesting direction for future investigation.\n\n\n>2. I found the connection with the three factor learning rule to be very interesting, but the abbreviated discussion made it a little hard to follow. In particular, it felt like the notation changed and it was not clear to me how Eq. 7 mapped onto the 3-factor learning rule. More on this would be helpful.\n\nWe thank the reviewer for pointing out the need for a clearer exposition regarding the connection to the three-factor learning rule. Eq. (7) expresses the gradient-based update derived from minimizing the layerwise decorrelation loss, but we acknowledge that the mapping to the classical three-factor form may not have been sufficiently transparent.\n\nTo clarify this connection, we note that Eq. (7) in more explicit form (with $\\vartheta$ substituted in to the equation) is given by\n\n$\n\\Delta W_{ij}^{(k)}[m] = \\zeta \\sum_{n = mB + 1}^{(m+1)B}\ng'^{(k)}_i(h^{(k)}_i[n]) \\cdot f'^{(k)}(u^{(k)}_i[n]) \\cdot q^{(k)}_i[n] \\cdot h^{(k-1)}_j[n]\n$\n\nwhen the batch size is set to \\( B = 1 \\), Eq. (7) simplifies to:\n\n$$\n\\Delta W_{ij}^{(k)} \\propto \ng'^{(k)}_i(h^{(k)}_i) \\cdot f'^{(k)}(u^{(k)}_i) \\cdot q^{(k)}_i \\cdot h^{(k-1)}_j.\n$$\n\nThis corresponds directly to the canonical three-factor learning rule form:\n$$\n\\Delta W_{ij} \\propto \\text{(post-activity)} \\times  \\text{(modulatory signal)} \\times \\text{(pre-activity)} ,\n$$ with the following identifications:\n\n  - **Pre-synaptic activity:**  $h^{(k-1)}_j$,\n  - **Post-synaptic activity:** $g'^{(k)}_i(h^{(k)}_i) f'^{(k)}(u^{(k)}_i)$,\n  - **Modulatory signal:** $q^{(k)}_i $, the projection of the output error onto the activations of layer $k$.\n\n\nIn the revised version, to make the connection between EBD update and three-factor learning rule more clear, we performed following changes: \n\n\n- in Section 2.1,  we included the EBD update expression for batchsize $B=1$: (Eq. (8))\n\n*\"For the special case of batchsize, $B=1$, the weight update in (7) simplifies to\n\n$$\n{\\Delta W_1^{(k)}[m]}_{ij} = \\zeta   {g'}^{(k)}_i(h_i^{(k)}[m]) {f'}^{(k)}(u_i^{(k)}[m]) q_i^{(k)}[m] h_j^{(k-1)}[m]. \n$$\n\"\n\n\n- we modified the the last sentence of the first paragraph of Section 3.1 to clarify the connection between EBD update and three-factor learning rule:\n\n*\"...   In contrast, EBD update for batchsize $B=1$ in (8) naturally matches the three-factor structure:...\"*\n\n- We also added a new figure  to Section 3, which describes EBD  learning as a three‑factor synaptic update. The following is the paragraph that refers to this figure:\n\n*\"Figure 2 breaks the EBD weight update (8) into its three interacting factors.  The presynaptic term is the activity $h_j^{(k-1)}$ from the sending unit; the postsynaptic term is $g_i^{\\prime(k)}f^{\\prime(k)}$ computed from the receiving unit’s own activation; and the modulatory term is the broadcast error $q_i^{(k)}$, derived from the network’s output error $\\boldsymbol{\\epsilon}$.  Multiplying these three quantities produces the weight change $\\Delta W^{(k)}_{ij}$ shown beneath the diagram, revealing that EBD naturally realises the classical three‑factor learning rule in deep networks.\"*\n\n\n>3. In Table 1, DFA + E (for MNIST) has nearly the exact same values as EBD, but it does not have its accuracy underlined (despite cases of actual ties). Is this because the performance is slightly different and EBD wins? Or is there maybe a typo on the performance of DFA + E?\n\n We thank the reviewer for this careful observation. The values for DFA + E and EBD on MNIST are indeed very close and we'll underline DFA +E values for MNIST and DFA value for MNIST-CNN  in the revised article's Table 1.\n\n>Minor points\n\n\n\n>1. I think Fig. 1 could be made bigger, to make it easier for the reader to see what the authors are showing.\n\nWe thank the reviewer for this helpful suggestion. We agree that enlarging Figure 1 would improve readability and help better illustrate the comparisons, particularly in panel Fig 1.c. In the submission, we kept the figure compact to reserve space for more discussion and theoretical development. However,  in the revised article, we increased its size so that some details and text are better visible.\n\n\n\n>2. In Eq. 3, there is the variable $R_{ge}^{(k)}$. Then on line 140, it is denoted as \n$R_{g^{(k)}(h(k)),e}$. Are those supposed to be the same thing or did I miss something?\n\nThanks for pointing out this typo. As the reviewer correctly noted these two different expressions are supposed to be same. We replace the wrong expression in line 140 (also after line 161, at line 163, Step 1 and Step 2 of Algorithm 1 and instances inside Algorithm 3 in Appendix I.3.2 ) with $R_{ge}^{(k)}$. \n\n---\n\n### Questions:\n\n>Why is the performance of DFA + E not underlined in Table 1?\n\nAs we noted in our earlier response,  the absence of underlining for DFA + E in Table 1 is due to an oversight in formatting and we will correct this in the revised version."}, {"review_id": "9oLdxZckMS", "reviewer": "Reviewer_Dubd", "summary": "This paper introduces a new theory for biologically plausible credit assignment that does not rely on backpropagation of the error. Established theories based on error broadcasting (EB) can avoid backpropagation but typically do not scale to difficult tasks or large networks. This paper introduces a new strategy termed EBD with is theoretically grounded in optimal MMSE estimators. It improves over existing EB strategies such as DFA as illustrated by thorough numerical simulation results. I congratulate the authors on these results and believe that this paper contributes both important conceptual and theoretical insights to the credit assignment literature.", "questions": "Questions:\n\n1. I understand how the EBD surrogate loss function can lead to collapse in the absence of activity regularization. However, I am curious to what extend the regularizers are *necessary*. Without regularization, do all networks eventually collapse? If so, how sensitive are the results to the choice of regularization hyperparameters? What was the reason to choose power normalization? \n2. Similarly, I am curious to what extend the method relies on the forward projections explained in Appendix C.5. How does removing these forward projections impact performance? \n3. How robust is EBD to different choices of the nonlinearity g(h)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- analytically grounded theory for learning with broadcasted errors. Setting the feedback weights to the  cross-correlation matrix between hidden activations output errors and the link to decorrelation adds a novel element to existing EB strategies.\n- Strong numerical results indicating that the method consistently outperforms DFA and could scale to larger networks. \n- The paper is generally well-written (although see below for comment on figures & notation).\n\nWeaknesses:\n\n- The results and model schematic could be illustrated better with some additional figures. \n- Mathematical notation, despite rigorous, is hard to follow at times and requires careful reading. This is excellent for the appendix but some expressions could be optimized for readability and clarity in the main text. \n- Some controls are missing, specifically with regards to the regularisation terms preventing collapse (see questions below)", "was_revised": true, "final_justification": "After reading the author rebuttal and reviews from other reviewers, I maintain that the method developed in this paper is a novel solution to error broadcasting that deserves attention and future investigations. \n\nEspecially if the authors address the minor issues with notations and presentation of the results, I recommend this paper for publication.", "rebuttal": "We would like to the reviewer for the constructive comments.\n\n\n### Strengths:\n\n>1. analytically grounded theory for learning with broadcasted errors. Setting the feedback weights to the cross-correlation matrix between hidden activations output errors and the link to decorrelation adds a novel element to existing EB strategies.\n\n>2. Strong numerical results indicating that the method consistently outperforms DFA and could scale to larger networks.\n\n>3.The paper is generally well-written (although see below for comment on figures & notation).\n\nWe really appreciate these positive comments.\n\n---\n### Weaknesses:\n\n>1. The results and model schematic could be illustrated better with some additional figures.\n\nWe agree informative schematics would aid clarity. In the revised article, we  added a new figure  to Section 3, which describes EBD  learning as a three‑factor synaptic update. The following is the paragraph that refers to this figure:\n\n*\"Figure 2 breaks the EBD weight update (8) into its three interacting factors.  The presynaptic term is the activity $h_j^{(k-1)}$ from the sending unit; the postsynaptic term is $g_i^{\\prime(k)}f^{\\prime(k)}$ computed from the receiving unit’s own activation; and the modulatory term is the broadcast error $q_i^{(k)}$, derived from the network’s output error $\\boldsymbol{\\epsilon}$.  Multiplying these three quantities produces the weight change $\\Delta W^{(k)}_{ij}$ shown beneath the diagram, revealing that EBD naturally realises the classical three‑factor learning rule in neural networks.\"* \n\nRegarding the experimental results, we have already included several plots of loss, accuracy, and correlation over training epochs in Appendix.  We included a new figure for correlation decay of CNN model in Appendix J.  We would be glad to add further result figures in the appendix, if the reviewer has specific suggestions.\n\n>2. Mathematical notation, despite rigorous, is hard to follow at times and requires careful reading. This is excellent for the appendix but some expressions could be optimized for readability and clarity in the main text.\n\nWe appreciate the reviewer’s constructive comment regarding the mathematical notation. In the revised version, we will further clarify  the notation in the main text to improve clarity and readability, while maintaining rigor. Specifically, we will unify notational conventions to eliminate redundant representations of the same variables, as also noted in other reviewer comments. Much of the perceived complexity arises from the explicit indexing over batch, sample, layer, and vector components, which we include for the sake of precision and reproducibility.  We  are  open to implementing simplifications that do not compromise these objectives and would welcome specific suggestions for improving readability. In order to simplify expressions, in the revised article, we provided batchsize $B=1$ versions of EBD update rule in (7) and power normalization loss in (Eq. (8) of initial submission).\n\n>3. Some controls are missing, specifically with regards to the regularisation terms preventing collapse (see questions below)\n\nWe address this comment through our response to the reviewer's questions below.\n\n---\n### Questions:\n\n>1. I understand how the EBD surrogate loss function can lead to collapse in the absence of activity regularization. However, I am curious to what extend the regularizers are necessary. Without regularization, do all networks eventually collapse? If so, how sensitive are the results to the choice of regularization hyperparameters? What was the reason to choose power normalization?\n\nWe appreciate the reviewer’s insightful question. As noted in the manuscript, the EBD surrogate loss admits a trivial minimizer corresponding to all-zero weights and collapsed representations. This necessitates the use of regularization mechanisms to ensure meaningful learning dynamics. To this end, we introduce two complementary regularization terms: entropy regularization and power regularization.\n\nThe entropy regularization encourages the use of the full representational capacity of each layer by promoting diversity in activations. However, this term alone can drive the network toward large activation magnitudes. The power regularization counterbalances this tendency by penalizing excessive activation energy, while also discouraging vanishing activations—thus playing a stabilizing role. Together, they maintain non-trivial, high-entropy yet bounded representations throughout training.\n\nWhile we have not conducted an exhaustive ablation study across all models and datasets, our preliminary experiments indicate that regularization significantly improves performance. For instance, in the MLP trained on CIFAR-10, removing both regularizers leads to a test accuracy drop from 55.5% to 53.1%. We also observe that entropy regularization benefits the DFA baseline, supporting the idea that it has broader utility beyond EBD.\n\nInterestingly, training with EBD does not  collapse even when regularizers are turned off—likely due to the use of non-zero, sufficiently diverse weight initializations. Yet, regularization makes the learning dynamics more robust and consistent.\n\nBeyond their algorithmic role, these regularizers also have biological interpretations: the power regularization can be viewed as a form of homeostatic control regulating the activations of neural activity, while entropy regularization aligns with the CorInfoMax framework, providing a potential normative explanation for the emergence of lateral connections in biological networks.\n\nA more detailed analysis of collapse mechanisms and regularization effects is an important direction for future work.\n\n>2. Similarly, I am curious to what extend the method relies on the forward projections explained in Appendix C.5. How does removing these forward projections impact performance?\n\nOur preliminary results suggest that the forward projections described in Appendix C.5 do not lead to significant performance gains—at least under the limited scope of our hyperparameter search. These projections were included only in the MLP experiments, and their associated weights were trained with very small learning rates (see Tables 3 and 4 in Appendix I.4.4), indicating a limited functional role during training. While theoretically motivated as a potentially useful mechansim, in practice their contribution appears marginal in the tested configurations. We plan to further investigate their impact in broader settings as part of future work.\n\n>3. How robust is EBD to different choices of the nonlinearity g(h)?\n\nWe appreciate the reviewer’s interest in the role of the nonlinearity $g(h)$ within EBD. Rather than viewing the choice of $g(h)$ as a potential vulnerability, we regard it as an opportunity to enhance performance, as an additional degree of freedom. While our current study primarily used simple form of  $g(h)=h$ and $g(h)=h^2$, we conducted preliminary experiments with alternative functions—such as sinusoidal nonlinearities—which yielded no signifcant performance improvement, though these investigations were not extensive.\n\nWe believe the choice of  $g(h)$ is an important design dimension that warrants deeper exploration. In particular, experimental studies on three-factor learning rules in biological systems may offer guidance on function classes that are optimized by nature. We see this as a promising direction for future work, aimed at both improving performance and drawing closer connections to biologically plausible computation."}, {"review_id": "dlpGdiZHc6", "reviewer": "Reviewer_Xg6E", "summary": "The authors present EBD (Error broadcast and decorrelation), a way to improve existing error broadcasting techniques (to address the credit assignment problem, i.e. how to assign importance - a different phrasing of \"how to train NNs\").\nIt works by altering the training objective to be layer-specific, and adjust network weights by lowering the correlation between their activations and the errors in broadcast outputs.\nThe authors ground their method with theoretic insights and test it on vision datasets, showing a PoC on small (MLP / Convolutional / Locally-connceted) networks using MNIST and CIFAR10.", "questions": "1. Why is a classification model trained using MSE? Is this standard practice in EB studies? Or is it something the authors employ for the MSEE orthogonality grounding?\n\n\n2. Are there any evidence that EBD or EB in general are biologically plausible, EXCEPT for their lack of requirement for weight symmetry?\n\n\n3. Suggestion: coming from the world of neural networks, I haven't witnessed the usage of the term \"credit assignment problem\" to weight learning. I think it would be best to drop this confusing phrasing from the abstract (despite it being explained in the third line of the intro - maybe leave it there).", "limitations": "yes (except for scale limitations discussed in the weaknesses section)", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Proper theoretical grounding and research method, with high relevance to its sub-field: The paper presents a new approach in the field of EB, grounds it in theory, and shows basic empirical results. This paper seems of high relevance to researchers working on this subject.\n\n2. Empirical improvement: While limited in terms of datasets, the benchmarking made by the authors shows an improvement over existing methods, making any further discussion of this method relevant to other works in this sub-field.\n\t\t\n\nWeaknesses: \n\n1. Not accessible to non-experts in this sub-field: As someone who is less familiar with the research work of Error broadcasting (also expressed in my low confidence score), I find it difficult that the paper isn't self-contained, and many parts of it still seem rather unclear to me. \nTo be more specific, I felt that this paper \"hits hard\" from the introduction --- it is very dense, throwing A LOT of new information on the motivation on EB and the theoretical background of the current work, without discussing much on how more basic EB work. I would've expected a preliminary section on EB *before* the authors dive in deep to their contributions in sec 2.2, which would further emphasize exactly what did the authors add *on-top* of existing EB work.\nAn additional example is the constant mentioning of the \"three-factor learning rule\" without defining what it is (having read what it is out of the paper, it is unclear to me why existing methods as backprop can't be considered as three-factor learning).\n\t\t \n2. Limited benchmarks: The results are only measured on MNIST on CIFAR10. The authors state this as a limitation, and while I *don't* hold it against the authors as they show a new direction in this work, I do think it would make sense to *explain* why the method wasn't (or can't be) tested on other simple datasets with similar resolutions to CIFAR10 (e.g., CIFAR100, Tiny imagenet, Caltech-256, etc), and not merely shrug it off as a \"direction for future work\". Even if other baseline results aren't available, seeing high EBD results on other datasets can create more faith in the method.\n\t\n3. Non-uniform / complex notation make it even more difficult to read: For instance, in eq 1 - u(k) should be mentioned as the pre-activation output; across the different equations, where Y is some vector, it is unclear if the indexing Y[m] or Y[mB] refers to a batch index (m) or an absolute index (mB).\n\t\t\nIn terms of explaining my rating:\nOverall, the first weakness (lack of accessibility to non-experts) is my major concern with this paper being accepted to the conference. I would also like to have a better intuition of weather this method scales to larger datasets (It's not clear it would scale easily, as I found some conflicting evidence in literature [1,2]). Nevertheless, because of its relevance to researchers within the field, I believe it adds value.\n\n\n[1] \"Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecture\", https://arxiv.org/abs/1807.04587\n\n[2] \"Biologically-plausible learning algorithms can scale to large datasets\", https://www.semanticscholar.org/paper/Biologically-plausible-learning-algorithms-can-to-Xiao-Chen/2c26cca611ef1403ca81df238d04d0ef4a584a86", "was_revised": true, "final_justification": "It seems to me (as a researcher from DL, definitely not an expert in the error brodcasting literature of this paper) that this paper is solid and deserves attention from the researchers in this sub-field. The authors responded to all my comments and have promised to revise the many points which I have found lacking. I therefore increase my score and think that this paper should be accepted.", "rebuttal": "We would like to the reviewer for the constructive comments.\n\n### Strengths:\n\nWe appreciate the positive comments by the reviewer.\n\n---\n### Weaknesses:\n>1. Not accessible to non-experts:... a preliminary section on EB ...  \"three-factor learning rule\" \n\nWe appreciate the feedback by the reviewer. Our work builds upon a growing body of research on error broadcasting (EB), and more broadly biologically plausible learning, and we recognize that its terminology and motivations may not yet be broadly familiar outside the immediate subfield. Given the length limitations of the conference and diversity of topics within the proposed framework, we were unable to provide in-depth introduction to the subject. Nonethless, based on the reviewer's feedback, in the revised article,  we updated the introduction section to improve its clarity for more general audience:\n\n- Added a concise preliminary description of basic error broadcasting in the paragraph (3) introducing it:*\"These methods involve broadcasting the global output error directly to all layers, often through random projections or fixed pathways, without relying on precise backward paths or symmetric weights\"*\n- Emphasized the contributions of EBD by noting that it builds on basic error broadcasting through layer-specific objectives grounded in estimation theory.\n- Defined the three-factor learning rule inline when first mentioned in the advantages paragraph, describing it as an extension of Hebbian plasticity that incorporates a neuromodulatory signal modulating updates based on pre- and postsynaptic activity. \n\nWe also note that Section 1.1 contains a survey paragraph on other EB methods, which we will further expand in the revision. We'll further go over the introduction and the other sections to improve clarity.\n\nMore specifically, regarding the connection between the standard backpropagation and three-factor learning rule: the backpropagation update expression can be put into a form similar to the three‑factor rule:\nFor a layer $l$ with weights $w_{ij}^{(l)}$, presynaptic activity $a_i^{(l-1)}$, membrane potential\n$z_j^{(l)}=\\sum\\nolimits_i w_{ij}^{(l)}\\,a_i^{(l-1)}$,  activation\n$a_j^{(l)}=\\phi\\!\\bigl(z_j^{(l)}\\bigr)$, and loss $L$, the gradient‑descent update factors as\n\n$$\\Delta W_{ij}^{(l)}=\n-\\eta\\,\\frac{\\partial L}{\\partial w_{ij}^{(l)}}=\n-\\eta\\,\n\\underset{\\text{pre-synaptic}}{a_i^{(l-1)}} \\cdot \n\\underset{\\text{post-synaptic}}{\\phi'\\bigl(z_j^{(l)}\\bigr)} \\cdot\n\\underset{\\text{error / third factor}}{\\varepsilon_j^{(l)}},\n$$\nwhere the back‑propagated error is defined recursively by\n$$\n\\varepsilon_j^{(l)}=\n\\sum_{k} w_{jk}^{(l+1)}\n\\delta_k^{(l+1)},\n$$\nwhere \n$$\n\\delta_k^{(l)}=\n\\phi'(z_k^{(l)})\n\\varepsilon_k^{(l)}.\n$$\n\nHowever, neuroscientists would not classify this as a true three‑factor rule. Its “third factor” is not a pure error signal but a neuron‑specific term combining higher‑layer activations and weights, computed by a separate error network whose synapses mirror those of the forward network—an arrangement viewed as biologically implausible. In biology, the third factor is a low‑dimensional, population‑wide neuromodulator delivered without such symmetric pathways.s [13].\n\nTo clarify BP and three-factor learning rule connection, as the second sentence in Section 3.1, we inserted the clarification sentence:\n\n*\"While backpropagation can be expressed similarly, it is not typically considered a three-factor rule in neuroscience, as its 'third factor' is a locally tailored signal specific to each neuron requiring a biologically implausible dual network with symmetric weights, unlike global neuromodulatory signals [13].    In contrast, EBD update for batchsize $B=1$ in (8)  naturally matches the three-factor structure:....\"*\n\nWe also included a new Figure in Section 3 to clarify three factor learning rule and its connection with EBD update. \n\n>2. Limited benchmarks:\n\nThis work primarily introduces a theoretically grounded learning framework. The numerical experiments are designed to validate the key concepts and demonstrate the performance advantages of our method over existing state-of-the-art error broadcast algorithms. We chose MNIST and CIFAR‑10 because they are the standard testbeds for broadcast‑based algorithms, and EBD already exceeds their state‑of‑the‑art results on both. \n\nWe further extended experiments to CIFAR‑100 using the same CNN architecture as in the Dropout paper [1], achieving 44.1% top‑1 accuracy—surpassing the original DFA result (41%) [2] and the best reported DFA result (43.7%) that we are aware of on this architecture [3]. In the revision, we will include these CIFAR‑100 results along with additional hyperparameter tuning.\n\nWe note that backpropagation itself once faced similar depth and data‑scale hurdles before architectural and optimization advances closed the gap; we can expect analogous innovations to further boost EBD performance on larger datasets going forward.\n\nRefs:\n- [1] Srivastava et al. \"Dropout: A Simple Way to Prevent Neural Networks from Overfitting\", JMLR, 2014.\n- [2] Crafton, Brian et al. “Direct Feedback Alignment With Sparse Connections for Local Learning.” Fron. in Neurosc. 2019.\n- [3] Nøkland, Arild. \"Direct feedback alignment provides learning in deep neural networks.\" Neurips, 2016.\n\n>3. Non-uniform/complex notation \n\nWe thank the reviewer for this helpful observation. We recognize that clear and consistent notation is critical for accessibility, and we will revise the manuscript to improve readability in response to this and related reviewer comments.\n\nIn particular, we will explicitly state that $\\mathbf{u}^{(k)}$ as preactivations at layer-$k$. This will be added after Eq. (1), along with a reminder that $k\\in\\{1, \\ldots L\\}$ is the layer index. \n\nRegarding indexing, as the reviewer correctly noted $m$ denotes the batch index and $mB+l$ is used to represent the absolute (sequence) index of the $l$-th sample within batch-$m$. This convention is used consistently throughout the mathematical derivations.\n\nTo further clarify:\n\n- Quantities such as weights $\\mathbf{W}^{(k)}[m]$, biases $\\mathbf{b}^{(k)}[m]$, learning rates $\\mu^{(k)}[m]$,  and projection matrices $\\hat{\\mathbf{R}}_{ge}^{(k)}[m]$ are updated per batch, hence indexed by $m$.\n- Batch-structured matrices such as $\\mathbf{E}[m]$, containing all output errors for batch-$m$, are similarly batch-indexed.\n\n To aid clarity, we will add the following in-line clarifications:\n \n - after Eq. (1): \"where $k\\in\\{1, \\ldots L\\}$ is the layer index, ...\"\n - at line 143: \"where $m$ is the batch index, $\\lambda\\in [0,1]$ is...\"\n - after Eq. (4): \"is the matrix of nonlinearly transformed activations of layer $k$ for batch $m$. In the above equation, $mB+l$ refers to absolute (sequence) index for the $l^{th}$ member of batch-$m$. Furthermore, \"\n\nIn addition, we will carefully review the manuscript for any further notational inconsistencies and perform minor revisions to improve uniformity and readability without compromising rigor.\n\n---\n### Questions:\n>1. Why  trained using MSE?\n\nAs the reviewer correctly noted, we use the MSE criterion due to its theoretical grounding in nonlinear MMSE estimation. However, experiments in Appendix J.2 show that correlations between output errors and layer activations still decay when training with cross-entropy loss. This suggests that the decorrelation principle, though derived from the MMSE objective, may extend to other losses. Exploring this theoretically is a promising direction for future work.\n\n>2.evidence EB biologically plausible?\n\nThis is a direction we plan to explore further. There is already encouraging evidence that error broadcast–based mechanisms may operate locally in various brain regions. Neuromodulators such as dopamine, serotonin, acetylcholine, and noradrenaline are strong candidates, as they project broadly across cortical and subcortical areas, delivering phasic signals that reach many neurons and synapses in parallel. This shared modulatory input can act as a global third factor in synaptic plasticity.\n\nReference [a] (ref. [13] in our paper) highlights specific examples: dopaminergic signals in the striatum, cholinergic and dopaminergic modulation in the hippocampus, and broader cortical regions—all supporting the role of global error-like signals in learning. Additionally, long-range projections to apical dendrites of pyramidal neurons may also support global error-projected learning.\n\nCollectively, these findings suggest that three-factor learning rules incorporating error broadcast are plausible biological mechanisms. Our work provides a normative framework for this hypothesis and aims to inspire further experimental and theoretical studies in biologically plausible error-driven learning.\n\n[a]. Gerstner, W. et.al. (2018). Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules. (Reference [13] )\n\n>Suggestion: ...\"credit assignment problem\"\n\nThank you for this  suggestion. We understand that the term “credit assignment problem” may not be as commonly used to describe weight learning in all sub-areas of machine learning. However, it has been used broadly in both computational neuroscience and biologically motivated machine learning literature to refer to the challenge of assigning learning responsibility to synapses or parameters. The usage of this terminology date backs to at least  Minsky and Papert's famous perceptron book [i], and it is still used frequenctly in neuro-AI papers (see the references in our article, e.g., [9],[14],[21], ..., and [ii] below). Since this term concisely captures the core problem addressed in our work and may help connect readers across disciplines, we would prefer to keep it in the abstract. \n\n[i] Minsky, M. & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry.\n\n[ii] Richards,& Lillicrap (2019). Dendritic solutions to the credit assignment problem. Curr. op. in neurobiology"}, {"review_id": "SxTtgFAlZf", "reviewer": "Reviewer_i7Bt", "summary": "The submission presents a new method for training neural networks that aims to minimize squared error without relying on standard backpropagation, by minimizing the correlation between the activations of individual layers and the network's residuals. Theoretical arguments based on minimum mean square error estimation theory are provided to justify this method. Experimental results show that the proposed method outperforms direct feedback alignment and global error vector broadcasting when classifying on MNIST and CIFAR-10, using MLPs, CNNs, and locally connected networks. However, classification accuracy remains lower than that achieved using backbropagation. The submission also shows that the proposed approach can be profitably applied in conjunction with more biologically plausible CorInfoMax networks.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The proposed approach is interesting, particularly considering the theory that supports it.\n\nAlthough theoretically motivated, the proposed algorithm seems to drop one of the two components of the derivative of the layer-specific loss without discussion of how this affects the theory.\n\nThe algorithm is based on squared error, but the experiments tackle classification problems. It seems that deep regression would be a more suitable application of the proposed method.\n\nThe experiments are limited to MNIST and CIFAR-10.\n\nThe hyperparameter search grids for the proposed method are different for MNIST and CIFAR-10.\n\nThe comparison to backpropagation is limited to fairly shallow networks.\n\nIt is not stated whether cross-entropy was used for backpropagation. If the squared error was used, this should be stated, and results for cross-entropy should be included.\n\nSome further related work that the authors should consider including in their discussion:\n\nWill Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algorithms can scale to large datasets. In International Conference on Learning Representations (ICLR), 2019.\n\nFrenkel, C., Lefebvre, M., & Bol, D. (2021). Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers in neuroscience, 15, 629892.\n\nMa, W. D. K., Lewis, J. P., & Kleijn, W. B. (2020, April). The HSIC bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 5085-5092).\n\nAkrout, M., Wilson, C., Humphreys, P., Lillicrap, T., & Tweed, D. B. (2019). Deep learning without weight transport. Advances in neural information processing systems, 32.", "was_revised": true, "final_justification": "I have raised my score from Borderline Accept to Accept after consideration of the rebuttal.", "rebuttal": "We would like to thank the reviewer for the constructive comments.\n\n### Strengths And Weaknesses:\n\n>The proposed approach is interesting, particularly considering the theory that supports it.\n\nWe appreciate the reviewer's positive remarks.\n\n>...the proposed algorithm seems to drop one of the two components of the derivative ....\n\nWe understand the reviewer’s concern. As described in the article, the omission of the second component is deliberate and the impact of this ommision is analyzed in Appendix E. In Section 2.3 and Appendix C, we first show that the decorrelation gradient naturally splits into two terms, $(\\Delta W_1,\\Delta b_1)$ that depend only on local activations and the broadcast error, and $(\\Delta W_2,\\Delta b_2)$ that would require backpropagating derivatives of the error (Appendix E.2). Keeping only $\\Delta W_1$ preserves strict locality. Our experimental results confirm that negative of this component essentially points in a descent direction: the cosine similarity of $\\Delta W_1$ with the \\emph{full} (two-term) gradient remains almost always positive throughout training (Fig. 4), confirming that -$\\Delta W_1$ is essentially a descent direction (Appendix E.2). Moreover, Appendix E.1 shows that $\\Delta W_1$ is also positively aligned with the backpropagation gradient (Figures 2 and 3). Furthermore, as demonstrated in Section 4, networks trained with the truncated update match or outperform state-of-the-art broadcast methods on all benchmarks. \n\n\n>The algorithm is based on squared error, but the experiments tackle classification problems. It seems that deep regression would be a more suitable application of the proposed method.\n\nWe thank the reviewer for this thoughtful observation. We agree that mean squared error (MSE) is a natural fit for regression problems where the target is continuous. However, our choice to evaluate on classification tasks was motivated by two main considerations: (i) the need for direct comparison with prior error broadcast–based methods, which predominantly evaluate on classification benchmarks; and (ii) the empirical observation that the core decorrelation mechanism underlying EBD remains effective even when cross-entropy loss is used.\n\nSpecifically, in Appendix J.2, we demonstrate that the decorrelation of layer representations still occurs under cross-entropy loss, suggesting that the surrogate EBD principle extends beyond the squared error setting. This helps explain the strong empirical performance we observe in classification tasks, despite the algorithm being theoretically grounded in MSE. We view this as an encouraging indication that the EBD framework may generalize to broader loss functions. Nonetheless, formalizing this generalization remains an important direction for future theoretical work.\n\n>The experiments are limited to MNIST and CIFAR-10.\n\nThis work primarily introduces a theoretically grounded learning framework. The numerical experiments are designed to validate the key concepts and demonstrate the performance advantages of our method over existing state-of-the-art error broadcast algorithms. We chose MNIST and CIFAR‑10 because they are the standard testbeds for broadcast‑based algorithms, and EBD already exceeds their state‑of‑the‑art results on both. \n\nTo further assess scalability, we extended experiments to CIFAR‑100 using the same CNN architecture as in the Dropout paper [1], achieving 44.1% top‑1 accuracy—surpassing the original DFA result (41%) [2] and the best reported DFA result (43.7%) that we are aware of on this architecture [3]. In the revised manuscript, we will include these CIFAR‑100 results along with additional hyperparameter tuning, and directly compare them to both existing DFA results in the literature and a standard backpropagation baseline.\n\n\nReferences:\n- [1] Srivastava et al. \"Dropout: A Simple Way to Prevent Neural Networks from Overfitting\", JMLR, 2014.\n- [2] Crafton, Brian et al. “Direct Feedback Alignment With Sparse Connections for Local Learning.” Frontiers in Neuroscience 13, 2019.\n- [3] Nøkland, Arild. \"Direct feedback alignment provides learning in deep neural networks.\" Neurips, 2016.\n\n\n>The hyperparameter search grids for the proposed method are different for MNIST and CIFAR-10.\n\nWhile the hyperparameter grids for MNIST and CIFAR-10 differ slightly, this is a common and justified practice due to the distinct nature and complexity of the two datasets. Our goal was to ensure that the proposed method operates competitively under reasonable settings for each benchmark. The search process remained consistent in scope and methodology across datasets.\n\n>The comparison to backpropagation is limited to fairly shallow networks.\n\nAs noted in the **Impact and Limitations** part of Section 5, our experiments with 10-layer CorInfoMax-EBD provide initial evidence that EBD has potential to scale to deeper architectures. While this demonstrates the depth scalability of the proposed framework, exploring broader scaling trends across architectures and input domains is an exciting direction for future work. We note that BP also initially struggled with challenges related to network depth and data scale, but these were eventually overcome through advances in architecture and optimization.  Similarly, we anticipate that similar innovations will further boost EBD performance on larger datasets going forward.\n\n>It is not stated whether cross-entropy was used for backpropagation. If the squared error was used, this should be stated, and results for cross-entropy should be included.\n\nBP was trained with MSE to maintain consistency with the MSE‑based objective used by the EBD algorithm. In the revised article, we will clarify that MSE criterion has been used for BP results. We'll also provide BP results with cross-entropy criterion. Our initial experiments with the cross-entropy loss yield test accuracy comparable to, or slightly better than, the MSE-based baseline—for example, 56.89% test accuracy for the MLP on the CIFAR-10 dataset, compared to 56.4% with MSE loss (see Table 1).  However, we will do broader hyperparameter sweep for the revision.\n\n>Some further related work that the authors should consider including in their discussion:\n\n>Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algorithms can scale to large datasets. In International Conference on Learning Representations (ICLR), 2019.\n\n>Frenkel, C., Lefebvre, M., & Bol, D. (2021). Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers in neuroscience, 15, 629892.\n\n>Ma, W. D. K., Lewis, J. P., & Kleijn, W. B. (2020, April). The HSIC bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 5085-5092).\n\n>Akrout, M., Wilson, C., Humphreys, P., Lillicrap, T., & Tweed, D. B. (2019). Deep learning without weight transport. Advances in neural information processing systems, 32.\n\nWe appreciate the reviewer’s relevant literature suggestions. In the revised article, we will expand Section 1.1 (Related Work and Contributions) by increasing the discussion on the previous work including the references pointed by the reviewer.\n\n---\n### Paper Formatting Concerns:\n>I did not spot any formatting issues. However, in my view, it is questionable whether a paper with such a substantial appendix is suitable for consideration at a conference. It is unreasonable to expect conference reviewers to check work of this size. This paper should have been submitted to a journal instead.\n\nWe thank the reviewer for the feedback. While we acknowledge that our appendix is longer than average, it is in line with many NeurIPS submissions—especially those involving theoretical developments or extensive experimental validations—which often include appendices exceeding ours in length. In our case, the appendix is intended to provide optional background material, full derivations, and additional empirical details for completeness and reproducibility.\n\nImportantly, we have taken care to ensure that the main paper is self-contained: all key ideas, contributions, and results are clearly presented within the 9-page main submission, without requiring the reader to consult the appendix.\nWe view NeurIPS as the most appropriate venue for presenting this framework to the research community, especially given that prior foundational work on error broadcast algorithms such as DFA and GEVB was also introduced at NeurIPS."}]

Author rebuttal:

Meta-review: The paper present a method for addressing credit assignment by error broadcasting wrt specific correlated layer activations.  The authors both ground their approach theoretically and demonstrate experimentally its effectiveness.  All reviewers agree on the quality of the paper and its expected impact.
Decision: Accept (spotlight)