Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry
Authors: Bariscan Bozkurt
Cengiz Pehlevan
Alper Tunga Erdogan
Venue: NeurIPS 2023 poster (2023)
Keywords: Correlative information maximization
Biologically-plausible learning
Multi-compartment neural model
TL;DR: We propose correlative information maximization as a normative supervised learning principle that allows derivation of biologically-plausible networks of pyramidal neurons and resolves the weight symmetry problem.

Abstract: The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks; however, its biological plausibility has been strongly criticized, and it remains an open question whether the brain employs supervised learning mechanisms akin to it. Here, we propose correlative information maximization between layer activations as an alternative normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This new framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. The coordinate descent-based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise to a neural network structure that emulates a more biologically realistic network of multi-compartment pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alternatives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing a compelling solution to this long-standing challenge.

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry
Bariscan Bozkurt 1223 Cengiz Pehlevan Alper T. Erdogan
"Gatsby Computational Neuroscience Unit, UCL, United Kingdom KUIS Al Center, Koe University, Turkey 'EEE Department, Koc University, Turkey 'John A. Paulson School of Engincering & Applied Sciences and Center for Brain Science, Harvard University, Cambridge, 02138 MA, USA
KKempner Institute for the Study of Natural and Artificial Intelligence
(bbozkurt15, alperdogan) ) @ku.edu. cpehlevan seas. harvard.
Abstract
The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks; however, biological plausibility has been strongly criticized, and it remains open question whether the brain employs supervised learning mechanisms akin it. Here, we propose correlative informa- tion maximization between layer activations as alternative normative approach to describe the signal propagation biological neural networks both forward and backward directions. This new framework addresses many concerns about the piauuiical of conventional artificial neural networks and the backpropa- gation algorithm. The coordinate descent- based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise neural network structure that emulates more bio- logically realistic network of multi-compartment pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, significant critique against the plausibility of the conven- tional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alterna- tives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing compelling solution this long-standing challenge.
biological plausibility
Introduction
How biological neural networks learn in supervised manner has long been open problem. The backpropagation algorithm [.. with its remarkable success training large-scale artificial neural networks and intuitive structure, has inspired proposals for how biologically plausible neural networks can perform the necessary efficient aasdinmessnn for supervised learning in deep neural architectures [2). Nonetheless, certain aspects of the backpropagation algorithm, combined with the oversimplified nature of artificial neurons, have been viewed as impediments to proposals rooted in this inspiration [3]
One of the primary critiques regarding the biological plausibility the backpropagation algorithm is the existence of parallel backward path for backpropagating error from the output towards the input, which uses the same synaptic weights as the forward path [ 2 4 Although such weight
transport, weight symmetry, is deemed highly unlikely based experimental evidence [3
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

some biologically plausible frameworks still exhibit this feature, which is justified by the symmetric structure of the Hebbian updates employed in these frameworks [2 6].
The concerns about the simplicity of artificial neurons have been addressed by models which incorporate multi-compartment neuron models into networked architectures and ascribe important functions to dendritic processing in credit assignment 7 8 2 10]. This new perspective has enabled the development of neural networks with improved biological plausibility.
In this article, we propose the use of correlative information maximization (CorInfoMax) among consecutive layers of neural network as new supervised objective for biologically plausible models, which offers
a principled solution to the weight symmetry problem: our proposed information theoretic crite- rion aims to maximize the linear dependence between the signals two neighboring layers, naturally leading to the use of linear or affine transformations in between them. A key property of this approach is that employing two alternative expressions for the correlative mutual information (CMI) results in potentially asymmetric forward and backward prediction networks, offering natural solution to the weight transport problem. Consequently, predictive coding in both directions emerges as the inherent solution to the correlative information maximization principle, fostering sig- nal transmission in both forward and top-down directions through asymmetrical connections. While the CorInfoMax principle enhances information flow in both directions, the introduction of set mem- bership constraints on the layer activations, such as non non-negativity, through activation nonlinearities and lateral inhibitions, encourages compression of information and sparse representations [I].
a normative approach for deriving networks with multi-compartment neurons: the gradient-based optimization of the CorInfoMax objective naturally leads to network models that employ multi- compartment pyramidal neuron models accompanied by interneurons as illustrated in Figure !|
As derived and explained in detail in Section 2 the resulting networks incorporate lateral connections and auto-synapses (autapses) to increase the entropy of layer, promoting utilization of all dimensions within the representation space of that layer. Meanwhile, asymmetric feedforward and feedback connections act forward and backward predictors of layer activation signals, respectively, to reduce the conditional entropies between layers, targeting the elimination of redundancy.
Ol++1
W)l
+ Hirdee
Figure 1: A segment of a correlative information maximization based neural network. Each layer consists of three-compartment pyramidal neurons with outputs r(()) and membrane voltages (u())_ soma, v -basal dendrites, v -distal apical dendrites) and interneurons with outputs ((k), The CMI
expression (1(55, r. (1++)) (F(0)) (((0))((()))))) ((++))) defines forward (backward) prediction synapses ((k)) if (w) ), for minimizing (ev) (zx+)))(())))))))))) (11(+)) (r(+))
(z())) (x+(++)))) and the lateral connections (k+1) (k)) and autapses D (k+1) (I()) connected to distal apical dendrites at the k + 1-th (k-th)
layer, for maximizing ((s)) (1(x)(r (((()))((())))))))

---

1.1 Related work
1.1.1
Multi-compartmental neuron model based biologically plausible approaches
Experimentally grounded studies, such as for Sendritic-processeee in multi-compartmenttl neurons for learning and credit assignment [13]. Subsequent research has explored biologically plausible models with supervised learning functionality, such the wwo-comarttment neuron model by Urbanczik and Senn [8] and the three-compartment
have been influential for considering a role
pyramidal neuron model by Sacramento et al. [9] Both models integrate non-Hebbian learning and spike-time dependent plasticity, while the latter includes SST interneurons [4] Similar frameworks have heen proposed by 15 and [TO), with the latter introducing a normative framework based on nulti-compartmental neuron structure, top-down feedback, lateral and feedforward connections, and Hebbian and non-Hebbian learning rules, emerging from the optimization of prediction error objective with a whitening constraint on co-layer neurons.
In a similar vein to [0], we propose an alternative normative framework based on information maximization principle. In this framework, the aare--ommarrtment structure and associated forward, top-down and lateral synaptic connections stem from the maximization of CMI between adjacent layers, without the imposition of any whitening constraint.
1.1.2 Weight symmetry problem
A central concern regarding the biological plausibility of the backpropagation algorithm pertains to the weight symmetry issue: synaptic weights in the feedback path for error backpropagation are transposes of those used in the forward inference path [6). The requirement of tied weights in backpropagation questionable for physically distinct feedforward and feedback paths in biological systems, leading many researchers to focus addressing the weight symmetry issue.
Various strategies have been devised to address the weight symmetry issue. For example, the feedback alignment approach, which fixes randomly initialized feedback weights and adapts feedforward weights, was offered as a plausible solution [7]. Later Akrout et.al. [T8] proposed its extension
by updating feedback weights towards to the transpose of the feedforward weights. Along the similar
lines, Amit introduced antisymmetry through separate random initializations [19]. Liao et al. [20] showed that the sign of the feedback weights (rather than their magnitude) affects the learning performance, and proposed the sign-symmetry algorithm.
Intriguingly, this symmetric weight structure is also observed in biologically plausible frameworks such as predictive coding
22 (23), equilibrium propagation (EP) [4 25 26]. and similarity matching [27]. This phenomenon can be rationalized by the transpose symmetry of the Hebbian update with respect to inputs and outputs. The EP framework in 25 unties forward and backward connections inspired by [28, 29), and only yields small performance degradation. A more recent approach by Golkar al. [00 addresses this challenge by integrating two alternative forward prediction error loss function terms associated with the same network layer and leveraging presumed whitening constraints to eliminate shared feedback coefficients.
(PC) [1] 22
In existing predictive coding- based schemes such as [21 22 23). the loss function contains only for- ward prediction error terms. The feedback connection with symmetric weights, which backpropagates forward prediction error, emerges due to the gradient- based optimization of the PC loss. In contrast, our framework's crucial contribution is the adoption of two alternative expressions for the correlative mutual information between consecutive network layers as the central normative approach. Utilizing these two alternatives naturally leads to both forward and backward prediction paths with asymmetric weights, promoting information flow in both feedforward and top-down directions. Unlike the work of [0), our method circumvents the need for layer whitening constraints and additional forward prediction terms to achieve asymmetric weights.
1.1.3 Correlative information maximization
Information maximization has been proposed as governing or guiding principle in several machine learning and neuroscience frameworks for different tasks: (i) The propagation of information within
a self-organized network as pioneered by Linsker [30]. (ii) Extracting hidden features or factors
associated with observations by maximizing information between the input and its internal represen-
tation such as independent component analysis (ICA-InfoMax, approach by [31. In the neuroscience
3

---

domain, the motivation has been to provide normative explanations to the behaviour of cortical activities evidenced by experimental work, such as orientation and visual stimuli length selectivity The same idea has been recently extended in the machine learning field by the Deep Infomax approach where the goal is to transfer maximum information from the input of a deep network to its final layer, while satisfying prior distribution constraints on the output representations [34]. (iii) Matching representations corresponding to two alternative augmentations modalities of the same input in the context of self-supervised learning (35).
of primary visual cortex neurons [3 3
Correlative mutual information maximization has been recently proposed as an alternative for Shannon Mutual Information (SMI), due to its desirable properties [36]: (i) maximization of CMI is equivalent to maximizing linear dependence, which may be more relevant than establishing arbitrary nonlinear dependence in certain applications [37] (ii) it is based only on the second order statistics, making it relatively easier to optimize. We additionally note that criteria based on correlation are intrinsically
linked to local learning rules, leading to biologically plausible implementations, [38 [9]. Erdogan 36 proposed the use of CorInfoMax for solving blind source separation (BSS) problem to retrieve potentially correlated components from their mixtures. Ozsoy et al. (37] proposed maximizing the
CMI between the representations of two different augmentations of the same input as a self-supervised learning approach. More recently, Bozkurt et al. [I] introduced an unsupervised framework to generate biologically plausible neural networks for the BSS problem with infinitely many domain selections using the CMI objective.
In this article, we suggest employing the CorInfoMax principle for biologically plausible supervised learning. The key difference compared to the unsupervised framework presented in [1] is the utilization of two alternative forms of mutual information. This leads to bidirectional information flow that enables error backpropagation without encountering the weight symmetry issue.
2 Deep correlative information maximization
2.1 Network data model
We assume a dataset with L input data points x[[] E I,..., L, and let YT [t] E R" be the corresponding labels. We consider a network with P hidden layers whose activities are denoted For notational simplicity, we also denote input and output of the network by (0) and p(P), i..., r [ [t] x(t) and p(P)(t) = Q(t). We consider polytopic constraints for the hidden and output layer activities, i.e., (8) E p((), where p(k) the presumed polytopic domain for the k-th layer [1]] 40). We note that the polytopic assumptions are plausible as the activations of neurons in practice are bounded. In particular, we will make the specific assumption that
by (*) IR Nx k 1,.... P
((k)_ Bx (r : 0 & & 1]. i.e., (normalized) activations lie in a nonnegative unit-hypercube. Such nonnegativity constraints have been connected to disentangling behavior [4] 42 43) however, we consider extensions in the form of alternative polytopic sets corresponding to different feature priors [I] (see Appendix C.. More broadly, the corresponding label yr can be, one-hot encoded label vectors for classification problem, or discrete or continuous valued vectors for regression problem.
2.2 Correlative information maximization based signal propagation
Our proposed CorInfoMax framework represents a principled approach where both the structure of the network and its internal dynamics as well as the learning rules governing adaptation of its parameters are not predetermined. Instead, these clements emerge naturally from an explicit optimization process. As the optimization objective, we propose the maximization of correlative mutual information (see Appendix A between two consecutive network layers. As derived in future sections, the proposed objective facilitates information for the hidden and output layers inherently induce information compression and feature shaping.
wnt--niputttooouppeee and vice versa, while the presumed domains
In Sections 2 and 2.2.2 we outline the correlative mutual information-basee objective and its implementation based on samples, respectively. Section 2.3 demonstrates that the optimization of this objective through gradient ascent naturally results recurrent neural networks with multi- compartment neurons. Finally, Section Section[.44explee e.4exinisss how the optimization of the same criterion leads to biologically plausible learning dynamics for the resulting network structure.

---

2.2.1 Stochastic CorInfoMax based supervised criterion
We propose the total correlative mutual information among consecutive layers, augmented with the mean meean-squae-ems (MSE) training loss, as the stochastic objective be maximized:
P-1
_ E(lyT - -1(
J(
(P))
Σ ((w (r(0)
((++))
(1)
K=0
where, as defined in [36 37
and in Appendix A
((re (x(()) p 1)+ log det (Ra(+) cxl) log det R Rumn- + cal)
1
(2)
2
is the correlative mutual information between layers p(h) an r(k+1), R E(r(kt))
is the autocorrelation matrix corresponding to the layer ((++1) activations, and R (k+1)) corresponds
to the error autocorrelation matrix for the best linear regularized minimum MSE predictor of (k++1)
from p(k), Therefore, the mutual information objective in (2) makes a referral to the regularized forward prediction problem represented by the optimization
minimize E(I] w
(B:) + ck W
fs 1|?
s.t.
- (k++)) = (k+1)
( ff (
(3)
(k+1) is the forward prediction error corresponding the optimal forward predictor W and e.
$1.**
If we interpret the maximization of CMI in (2): the first term on the right side of ( encourages the spread of p(++1) in its presumed domain p(++1), while the second term incites the minimization of redundancy in (k++1) beyond its component predictable from (kk)
An equal and alternative expression for the CMI can be written as (Appendix A
1 T(az) (k+11) 2
log det (R,c0 ex log det (R_ + ex ex
where (k corresponds to the error autocorrelation matrix for the best linear regularized minimum MSE predictor of (()) from (k++)), The corresponding regularized backward prediction problem is defined by the optimization
minimize E([l (k) with
(
c
W (8) 11
((k) (A())
(7()))((++)) fb
We observe that the two alternative yet equivalent representations of the correlative mutual informa- tion between layers (8) and ((++1) in (2) and (4) are intrinsically linked to the forward and backward prediction problems between these layers, which are represented by the optimizations in 33 and (5) re- spectively. As we will demonstrate later, the existence of these two alternative forms for the CMI plays crucial role in deriving neural network architecture that overcomes the weight symmetry issue.
2.2.2 Sample-based supervised CorInfoMax criterion
Our aim is to construct a biologically plausible neural network that optimizes the total CMI, equation [ in an adaptive manner. Here, we obtain a sample- based version of [I as a step towards that goal.
We first define the exponentially-weigheed sample auto and cross-correlation matrices follows:
Re,
/ [4]
(+)) [T)"" (((0))( [4]
A
(K) (1++1))
(6)
for i 0...., equivalent forms of the sasele-based CMI, fle) (r((), ((++)))(t]
,, P. respectively, where < X < is the forgetting factor. Next, we define two
f(av (I(M, (k++)))
1
1
2 log det det(R, ( (R, + I)
log det R [ cxI),
(7)
2
f((t) [((+1))[E] log det R, + E
1
log det R_ + +
(8)
2
2
e.
5

---

where R Rg is the exponentiall-wwiighee sample autocorrelation matrix for the forward
- (k+1), [1] corresponding to the best linear exponentially-weighhed prediction error at level-(k: + 1), e
regularized least squares predictor of (()+)((t) from the lower level activations [t) Similarly, autocorrelation matrix for the backward prediction error at
R. (o [t] is the exponentially-weigghh
(Ak level-(k), e
[t]. corresponding to the best linear exponentially-wegghes regularized least squares predictor of r ((()() [t] from the higher level activations [t)
The sample-basee CorInfoMax optimization can be written as:
3 2 Ilvrlt rr( j(
maximize
(r() p
r (1) [0 -0,
P....,, P
k=
(9a)
subject to
[t] [t)
pl(k K I.......
(9b)
[0)(t x [t)
(9c)
As outlined in Appendix B we can employ Taylor series linearization to approximate the log det terms associated with forward and backward prediction errors in (2) and (4) in the form
log det
R
R [f] + cxT))
x -
107,000000 a ( f + No+1 log(e:)
(10)
CK
1=1
det
det (0 ERI)
Ck (100
w(i)) jb, (1)
ex [
fa. 141) + N₂ log(ex),
(11)
ff. where W [t] is the optimal linear regularized weighted least squares forward predictor coefficients
in predicting r (()))((i from (()( [1] for 1,...,4, and (((k fb. [4] is the optimal linear regularized weighted least squares backward predictor coefficients in predicting [() from ((++)( for =
1.....t. Consequently, the optimal choices of forward and backward predictor coefficients
coupled with the optimal choices of layer activations.
In the online optimization process, we initially relax the requirement on the optimality of predictors and start with random predictor coefficient selections. During the learning process, we apply coordinate ascent-based procedure on activation signals and predictor coefficients. Specifically, at time step-t, we consider two phases:
1. First, optimize with respect to the activations [t. P), where assume predictor coefficients to be fixed. This phase yields network structure and output dynamics,
2. Next, we update the forward and backward predictor coefficients with and (
fb for k = 1. 1....,,, P. to reduce the corresponding forward and backward prediction errors, respectively. This phase provides update expressions be utilized in learning dynamics.
As the algorithm iterations progress, the predictor coefficients converge to the vicinity of their optimal values.
For the first phase of the online optimization, employ projected gradient ascent-based approach for activations: for 1,..., P 1, the layer activation vector [())(t is included in the objective function terms j(() (x(b-1), [() and j(t) (r()) [(*+1))()] [d]. Therefore, to calculate the gradient with
respect [1)) we can use expressions in and (8) More specifically, we choose (r =
(x(t))) (()) + f(s)) r for 1,..., P 1, to represent the components of the objective function in Dab involving (()) described Appendix G. this choice is instrumental in avoiding weight transport problem. Similarly, can write the component of

---

the objective function in 9a that is dependent on the final layer activations (r))
Ker ( (P)
[c]
2
I
yr[tl
Based on the derivations presented in Appendix [D] which directly incorporate the approximations
from 10 and
(11),
express the gradient of the objective function in (9a) with respect to r())
for k
1 as:
V pas J(n I(
[1] =2;
2; Brcss (1)0000000
1
[1]
(k)
[0]
(12)
ck-1
Ck
where 7
1_de,
2,
(1) [t] = =r(k((t) - 5s
( (1)) (t),
[t] = 1 ((h) =²h0
è
w [t) [t),
fb
(13)
and B,(+)(d) [1] (R, + [t] + ck- I] 11 ~ (R [1] ck I Similarly, for P. we have
J(()) ,.... p()) it] BB (t))
c [t] (r yr(l))
(14)
CP-1
2.3 Neural network formulation based on information maximization
In this section, we develop a biologically plausible neural network grounded on the correla- tive information maximization-basel network propagation model outlined in Section 2.2 To achieve this, we employ projected gradient ascent optimization for determining layer activations (()))) (()))()....... r.,,.....,.... (t)) [tt.. which shape the network structure and dynamics, as well as updating the corresponding synapses that govern the learning dynamics.
2.3.1 Network structure and neural dynamics
In this section, we show that the projected gradient ascent solution to the optimization in defines a multilayer recurrent neural network. To this end, we introduce the intermediate variable u (*)
as the updated layer-k activations prior the projection onto the domain p(k), Utilizing the
gradient expressions in (12)- 13), we can express the network dynamics for layers k = 1,... P
follows (see Appendix B details):
du (k) (1(
1
1 (k) [t:s]
(k) [t;
Tu
u [;:s] + - MM (t)
(A)
eu
(15)
ds
EK
Ek-
Ek
e, [t: = (t: s]
(B)
(1) [1;ss)
e, [] (1 s]
whb
[t;
ff
fb
(16)
[11)) a 00 (t:)))
(17)
where is the discrete data index, is the continuous time index corresponding network dynamics, Tu is the update time constant, (*) cu (2 [t] + gueT,, and 0 represents the elementwise clipped-ReLA ReLU function corresponding to the projection onto the nonnegative unit-hypereube B
defined as a (u) min(1, max(u, 0)).
To reinterpret the dynamics in (15 to as a multi-comppatmenntll neural network, for k == P - 1, we define the signals:
(k) [1; s] M (1) [1] s] + w [(]](++)) s],
(k) [t: [t:
w fs
[ (1-1)) [t:s], (18)
which allow us to rewrite the network activation dynamics
15 to as:
du (k) st - (8) [t: s] + 9. [[]ts]] [t:
Tu
(8) ( [t: s)) + IB, (v
st
u
[t;s))) (19)
r (1)
I;: (u s)),
(20)
where gA.k = I and gA.k 1. Similarly, for the output layer, we employ the same expressions
as
19 and
220 with & = P. except that in this case we have:
[t s] = M (P)
(x s] yr(t)), vB
(P) [t:s] wif
(P-1)) (tp-"(t (P1)) (21)
7

---

where 9B.P
1 cp-1 9A.P 3 and M (t = 5 5 [ + I.x
Remarkably, the equations 18 to 21 reveal biologically plausible neural network that incorporates
compartmmtt pyramidal neuron models, as presented | [0]. This intricate architecture, of which two-layer segment is demonstrated in Figure I naturally emerges from the proposed correlative information maximization framework. In this network structure:
(()) embodies the membrane potentials for neuronal somatic compartments of the neurons at layer-k; where Tu is the membrane leak time constant of soma.
v/ corresponds membrane potentials for basal dendrite compartments, receiving feedforward input originating from the previous layer.
A denotes the membrane potentials for distal apical dendrite compartments, which gather top- down input from the subsequent layer and lateral imputs represented by ( [t] in 18 and (11 Decomposing (k) into ( (k), we find that (*) mirrors autapses [4], and the off-diagonal component O (&) corresponds to lateral inhibition synapses. We use i(k) - (k)) to represent the activations of SST interneurons
(k)
[4 that generate lateral inhibitions the apical dendrites.
Forward (backward) prediction errors manifest the membrane voltage differences between soma and basal (distal) compartments of the pyramidal neurons.
(top-down) synapses connecting layers (R)) and (k ff (W(b)
Forward (backward) prediction coefficients W (k)
associated with feedforward
( 1).
The inverse of the regularization coefficient Ej related to the conductance between soma and den- dritic compartments. This is compliant with the interpretation of the e in Appendix (A.2 as the sen- sitivity parameter that determines the contribution the prediction errors the CMI. Conversely, at the output layer, the augmentation constant / corresponds to the conductance between soma and distal compartments. This relationship can be motivated by modifying the objective in (9a) as
ilex) (m(h)) ((++1))(+
1 ji0 (r(P)) .y)((1)
(22)
2
where, through the first-order approximation. the [() dependent portion of i yT)(( can be expressed as (P) w/ (P) yr [t]1]] For accuracy, enforce W (P) I.
fb
2.4 Learning dynamics
ff feedback ( Network parameters consists of feedforward W (R)
fb and lateral (k) coefficients. The
learning dynamics of these coefficients are elaborated below:
Feedforward Coefficients are connected to the forward prediction problem defined by the op- timization in (3). We can define the corresponding online optimization objective function as Crr(W(/) (k) ex| [ (k) ff ll (*+11) [1]123 for which the the partial derivative is given by
(k+1)
aC, is (t) = 2cp w M
(k)
( (H]"""
2e - (k+1)
(23)
aw!
Appendix we provide a discussion on rewriting
233 ) in terms of the membrane voltage
difference between the distal apical and soma compartments of the neuron, based on the equilibrium condition for the neuronal dynamics:
- (1++1) (*) itjT = g (OAAA (1 - (gux + 9A. )u
(4) [t] h. (+))
(24)
where h.[t] is nonzero only for neurons that are silent or firing at the maximum rate.
Similarly, Feedback Coefficients are connected to the backward prediction problem defined by the
optimization in 55. and the corresponding online optimization objective function as Cya( (W(()))
(l)
< A /f 1p? + l
[t] for which the partial derivative is given by
W 18 [4]
2cp W (8)) 20 (A) (1]
(25)
aw(h) (k)
8

---

To compute the updates of both feedforward and feedback coefficients, we use the EP approach 24 where the update terms are obtained based on the contrastive expressions of partial derivatives
23
and
25 for the nudge phase, i.e., 3 = B > 0, and the free phase, i.e., 8 = 0.:
16 11 Ow
(A+1)
[1 (1)))
B=9" (e
(k+1)
(26)
B
B=0
ow
10) fb
[t]
1 B'
(e
(K)
('e (k)
[t]
(27)
Lateral Coefficients, (()) are the inverses of the cI perturbed correlation matrices. We can use the update rule in [ for their learning dynamics after the nudge phase:
B [t+ x
(B
7z (k) (t)))) where (k) [t) 8-
(28)
As we derived in Appendix F can rewrite the update rule of the lateral weights in terms of the updates of autapses and lateral inhibition synapses as follows:
D | D 3 - a. (z Eagin ) Ei [I....,NNs) No} (29)
[ = o (*) + (h) 112 [1, Vi,j E fI,...,,M.,, Nu), where i f (30)
3 Discussion of results
In A.14), we devise an update for layer activation (8) by employing two distinct forms of the CMI associated with r(A); j(ee-> (r [()) the CMI with the preceding layer, encompassing
the forward prediction error for estimating r(*), and (r))) D)t), the CMI with the subsequent layer, incorporating the backward prediction error for estimating (kk), Employing these alternative expressions is crucial in circumventing the weight transport problem and offering more biologically plausible framework. For further discussion, please refer to Appendix G
In the context of the proposed correlative information maximization framework, forward and backward predictive coding naturally emerges as a crucial mechanism. By incorporating both alternative expressions of CMI, the framework focuses on minimizing both forward and backward prediction errors between adjacent layers via feedforward and feedback connections. These connections foster bidirectional information flow, thereby enhancing the overall learning process.
I depicts the interplay between the CorInfoMax objective and the corresponding network architecture. The emergence of lateral connections and autapses can be attributed to the maximization of the unconditional layer entropy component of the CMI, which allows for efficient utilization of the available representation dimensions and avoids dimensional degeneracy. Simultaneously. the minimization of conditional entropies between adjacent layers gives rise to feedforward and feedback connections, effectively reducing redundancy within representations.
Figure
We employ time-contrastive learning, as in GenRee [45], EP [24) and CSM (27), by implementing separate phases with Hebbian and anti-Hebbian updates, governed by an assumed teaching signal. It has been conjectured that the teaching signal in biological networks can be modeled by the oscillations in the brain in the brain are elusive, they are believed to play an important role in adaptive processes such as learning and predicting upcoming events [48 49
46 47 Although the oscillatory rhythms and their synchronization
4 Numerical experiments
In this section, we evaluate the performance of our CorInfoMax framework with two layer fully connected networks on image classification tasks using three popular datasets: MNIST [50]. Fashion-MNIST [51), and CIFARIO [52]. We used layer sizes of 784, 500, 10 for both MNIST and Fashion-MNIST datasets while we used layer sizes of 3072, 1000, 10 for CIFAR 10 dataset, and the final layer size corresponds to one-hot encoded ouput vectors. Further details including full set of hyperparameters can be found in Appendix J We compare the effectiveness of our approach against

---

other contrastive methods, such as EP 24 and CSM 127. as well as explicit methods, including PC 22 and PC-Nudge [53], when training multilayer perceptron (MLP) architectures.
We examine two distinct constraints on the activations of CorInfoMax Networks: (i) B ing the nonnegative part of the unit hypereube, and (ii) B1,+ {r r > O, ||||| 1}, denoting the nonnegative part of the unit -n-nrrm ball [40]. Table pressenss the test accuracy results for each algo- rithm, averaged over 10 realizations along with the corresponding standard deviations. These findings demonstrate that CorInfoMax networks can achieve comparable or superior performance in relation to the state-of state-of-the of-the-art methods for the selected tasks. Additional information regarding these exper- iments, as well as further experiments, can be found in the Appendix. Our code is available online
+. represent-
Table 1:
Test accuracy results (mean + standard deviation from 10 runs) for CarInfoMax networks are compared with other biologically-plassin algorithms. The performance of CSM on the CIFARIO dataset is taken from 27. while the remaining results stem from our own simulations.
MNIST
FashionMNIST
CIFARIO
CorInfoMax--B...t (Appendix 1.4)
97.62 + 0.1
88.14 0.3
51.86 + 0.3
CarInfoMax-BI,, (Appendix [.6)
97.71 + 0.1
88.09 + 0.1
51.19 + 0.4
EP
97.61 + 0.1
88.06 + 0.7
49.28 + 0.5
CSM
98.08 = 0.1
88.73 = 0.2
40.79*
PC
98.17 #
89.31 0.4
PC-Nudge
97.71
88.49 = 0.3
48.58 + 0.7
Feedback Alignment (with MSE Loss)
88.72 0.5
50.75 + 0.4
Feedback Alignment (with CrossEntropy Loss)
97.95 - 0.08
88.38 + 0.9
52.37 + 0.4
BP (with MSE Loss)
97.58 + 0.01
88.39 0.1
52.75 + 0.1
BP (with CrossEntropy Loss)
98.27 = 0.03
89.41
53.96 + 0.3
5 Discussion and Conclusion
In this article, we have presented the correlative information maximization (CorInfoMax) framework as biologically plausible approach constructing supervised neural network models. Our proposed method addresses the long-standing weight symmetry issue by providing principled solution, which results in asymmetric forward and backward prediction networks. The experimental analyses demon- strates that CorInfoMax networks provide better or on-par performance in image classification tasks compared to other biologically plausible networks while alleviating the weight symmetry problem. Furthermore, the CorInfoMax framework offers a normative approach for developing network models that incorporate comparccomprttment pyramidal neuron models, aligning more closely with the exper- imental findings about the biological neural networks. The proposed framework is useful in obtaining potential insights such as the role of lateral connections in embedding space expansion and avoiding degeneracy, feedback and feedforward connections for prediction to reduce redundancy, and activation functions//ntermeuron. to shape feature space and compress. Despite the emphasis on supervised deep neural networks in our work, it's crucial to highlight that our hoproach-rreplacing the backpropaga- tion algorithm, which suffers from the weight transportation problem, with a normative method devoid of such issues-is potentially extendable to unsupervised and self-supervised learning contexts.
One potential limitation of our framework, shared by other supervised approaches, is the necessity for model parameter search to improve accuracy. We discuss this issue in detail in Appendix K Another limitation stems from the intrinsic nature of our approach, which involves the determination activities through recursive dynamics (see Appendix JJ. While this aspect is fundamental to methodology, it does result in slower computation times compared to conventional neural networks in digital hardware implementation. However, it is worth noting that our proposed network, characterized by local learning rules, holds the potential for efficient and low-power implementations on future neuromorphic hardware chips. Furthermore, method employs the time contrastive learning technique known as Equilibrium Propagation, which necessitates two distinct phases for learning.
neural
10

---

Acknowledgments and Disclosure of Funding
This research was supported by KUIS AI Center Research Award. B. Bozkurt acknowledges the support by Gatsby PhD programme, which is supported by the Gatsby Charitable Foundation (GAT3850). C. Pehlevan supported by NSF Award DMS-2134157, NSF CAREER Award IIS- 2239780, and Sloan Research Fellowship. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
[1] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating errors. Nature, (2366)8) 1986.
[2] James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in cognitive sciences, 23(3):235-22) 2019.
[3] Francis Crick. The recent excitement about neural networks. Nature, 337: 1989.
[4] Stephen Grossherg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-33, 1987.
[5] Xiaohui Xie and H Scbastian Seung. Equivalence of backpropagation and contrastive hebbian learning in a layered network. Neural computation, 15(2):441-454, 2003.
[6] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between eneryy-based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
[7] Matthew Larkum. A cellular mechanism for cortical associations: organizing principle for the cerebral cortex. Trends in neurosciences, 36(3):141-15,, 2013.
[8] Robert Urbanczik and Walter Senn. Learning by the dendritic prediction of somatic spiking. Neuron, 81()):221-528, 2014.
[9] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. Advances in neural information processing systems, 31, 2018.
[10] Siavash Golkar, Tiberiu Tesileanu, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. Constrained predictive coding as biologically plausible model of the cortical hierarchy. Ad- vances in Neural Information Processing Systems, 35:14155-1411 2022.
[11] Bariscan Bozkurt, Ateş isfendiyaroglu, Cengiz Pehlevan, and Alper Tunga Erdogan. Correlative information maximization based biologically plausible neural networks for correlated source separation. In The Eleventh International Conference on Learning Representations, 2023.
[12] Leopoldo Petreanu, Tianyi Mao, Scott M Sternson, and Karel Svoboda. The subcellular organization of neocortical excitatory connections. Nature, (137(733):1142-111 2009.
[13] Blake A Richards and Timothy P Lillicrap. Dendritic solutions the credit assignment problem. Current opinion in neurobiology, 54:28-36, 2019.
[14] Joanna Urban-Ciecke and Alison Barth. Somatostatin expresssiniiieeennneeeeeeeieiegg neurons in cortical networks. Nature Reviews Neuroscience, 17(7):401-409 2016.
[15] Jordan Guerguiev, Timothy Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. Elife, 6:222901, 2017.
[16] Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cogn. Sci., 11:23-63, 1987.

---

[17] Timothy P "Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support enror backpropagation for deep learning. Nature communications. 7(1):13276, 2016.
[18] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019.
[19] Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in computational neuroscience, 13:18, 2019.
[20] Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropa- gation? Proceedings of the AAAI Conference on Artificial Intelligence, 30, 10 2015.
[21] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-fieee field effects. Nature neuroscience, 2(1):79-87 87, 1999.
[22] James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algo- rithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5):1229-122 1262, 2017.
[23] Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the brain do backpropagation? exact implementation of backpropagation in predictive coding networks. Advances neural information processing systems, 13:22566-22579 2020.
[24] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, 2017.
[25] Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation deep convnets by drastically reducing its gradient estimator bias. Frontiers in Neuroscience, 15:633674, 02 2021.
[26] Axel Laborieux and Friedemann Zenke. Holomorphic equilibrium propagation computes exact gradients through finite size oscillations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[27] Shanshan Qin, Nayantara Mudur, and Cengiz Pehlevan. Contrastive similarity matching for supervised learning. Neural computation, (3(5):3000-125 2021.
[28] Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, and Yoshua Bengio. Generalization of equilibrium propagation to vector field dynamics, 2018.
[29] J.F. Kolen and J.B. Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN '94), volume 3, pages 1375-1380 vol.3, 1994.
[30] Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3): 1988.
[31] Anthony Bell and Terrence J Sejnowski. An informaiionmmaximizztioo approach to blind separation and blind deconvolution. Neural computation, 7(6): 119-119,, 1995.
[32] D.H. Hubel and T.N. Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of physiology, 148:574-591, 1959.
[33] Authony Bell and Terrence J Sejnowski. The "independent components" of natural scenes are edge filters. Vision research, 37(23)33327-333 1997.
[34] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildans Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations (ICLR), 2019.
12

---

[35] Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces random-de stereograms. Nature, 355(6356)::1111163, 1992.
[36] Alper T Erdogan. An information maximization based blind source separation approach for dependent and independent sources. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4378 4382, 2022.
[37] Serdar Ozsoy, Shadi Hamdan, Sercan O Arik, Deniz Yuret, and Alper Tunga Erdogan. Self- supervised learning with an information maximization criterion. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[38] Erkki Oja. Simplified neuron model principal component analyzer. Journal of mathematical biology, 15:267-273, 1982.
[39] David Lipshutz. Yanis Bahroun, Siavash Golkar, Anirvan Sengupta, and Dmitri Chklovskii. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation, 33(9):2309-233 2352, 2021.
[40] Gokean Tatli and Alper T Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing, 69:5431-5447, 2021.
[41] Mark D Plumbley. Algorithms for nonnegative independent component analysis. IEEE Trans- actions on Neural Nenworks, 14(3):534-543, 2003.
[42] Cengiz Pehlevan, Sreyas Mohan, and Dmiuri B Chklovskii. Blind nonnegative source separation using biological neural networks. Neural computation, 29(11):2225-2954 2017.
[43] James C. R. Whittington, Will Dorrell, Surya Ganguli, and Timothy Behrens. Disentanglement with biological constraints: A theory of funetional cell types. In The Eleventh International Conference on Learning Representations, 2023.
[44] Joachim Lübke, Henry Markram, Michael Frotscher, and Bert Sakmann. Frequency and dendritic distribution of autapses established by layer 5 pyramidal neurons in the developing rat neocortex: comparison with synaptic innervation of adjacent neurons of the same class. Journal of Neuroscience, 16(10):320933218 1996.
[45] Randall O' O'Reilly. Biologically plausible error-driven learning using local activation differ- ences: The generalized recirculation algorithm. Neural computation, ((5)88959988, 1996.
[46] Pierre Baldi and Fernando Pineda. Contrastive learning and neural oscillations. Computation, ((4):526545,, 1991.
Neural
[47] Nicholas Ketz, Srinimisha G Morkonda, and Randall CO' Theta coordinated error-driven learning in the hippocampus. PLoS computational biology, 96))e1003067, 2013.
[48] Juergen Fell and Nikolai Axmacher. The role of phase synchronization in memory processes. Nature reviews. Neuroxcience, 12:105- 02 201
[49] Andreas Engel, Pascal Fries, and Wolf Singer. Dynamic predictions: oscillations and synchrony in top-down processing. Nature reviews. Neuroscience, 2:704-716, 11 2001.
[50] Yann LeCun and Corinna Cortes.
MNIST handwritten digit database.
http://aan..eeuu.co//xx//mmmmmm 2010.
[51] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
[52] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
13

---

[53] Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. Backpropagation at the infinitesimal inference limit of energy-based models: Unifying predictive coding, equilibrium propagation, and contrastive hebbian learning. In The Eleventh International Conference on Learning Representations, 2023.
[54] Zhanghao Zhouyin and Ding Liu, Understanding neural networks with logarithm determinant entropy estimator. arXiv preprint arXiv::401.342 2021.
[55] Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear estimation. Prentice-Han information and system sciences series. Prentice Hall, 2000.
[56] Xiao Fu, Kejun Huang, Bo Yang. Wing-Kin Ma, and Nicholas D Sidiropoulos. Robust volume minimization-basee matrix factorization for remote sensing and document clustering. IEEE Transactions on Signal Processing, 64(23):6254-622 2016.
[57] Arne Brondsted. An Introduction to Convex Polytopes, volume 90. Springer Science & Business Media, 2012.
[58] David L Donoho. For most large underdetermined systems of equations, the minimal & nommo near-solution approximates the sparsest near-solution. Communications on Pure and Applied Mathematics, 59(7):907-934, July 2006.
[59] Michael Elad. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer Science & Business Media, 2010.
[60] John Duchi, Shai Chalev-Shwartz Yoram Singer, and Tushar Chandra. Efficient projections onto the ball for learning in high dimensions. In Proceedings of the 25th International Conference Machine learning, pages 272-279, July 2008.
[61] Eren Babatas and Alper T Erdogan. An algorithmic framework for sparse bounded component analysis. IEEE Transactions on Signal Processing, 66(19):5194-520) August 2018.
[62] Christoph Studer, Tom Goldstein, Wotao Yin, and Richard G Baraniuk. Democratic representa- tions. arXiv preprint arXiv: 1401.3420, 2014.
[63] Alper T Erdogan. A class of bounded component analysis algorithms for the separation of both independent and dependent sources. IEEE Transactions on Signal Processing, 6122):5773- 5743, August 2013.
[64] Huseyin A Inan and Alper T Erdogan. A convolutive bounded component analysis framework for potentially nonstationary independent and/or dependent sources. IEEE Transactions on Signal Processing, 63(1::18-30, November 2014.
[65] Bariscan Bozkurt, Cengiz Pehlevan, and Alper Tunga Erdogan. poologically-plassibbe deter- minant maximization neural networks for blind separation of correlated sources. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[66] Christopher J Rozell, Don H Johnson, Richard G Baraniuk, and Bruno A Olshausen. Sparse cod- ing via thresholding and local competition in neural circuits. Neural computation, 20(10):2526- 2563, 2008
[67] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[68] Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends@ in Optimiza- tion, 1(3):127-239, 2014.
[69] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035 Curran Associates, Inc., 2019.
14


Reviews:
[{"review_id": "VvcQ0iVtOn", "reviewer": "Reviewer_MCJo", "summary": "The paper proposes Correlative Information Maximization as an underlying objective for biologically plausible learning. The objective produces a multi-compartmental neuron model, and can operate with feedback connection that are plastic, but not tied to the feedforward ones. ", "questions": "Main question: are the feedback weights different from the feedforward at the end of training? I think it is an interesting question in itself, since \"no\" would mean your algorithm approximates backprop (similar to predictive coding approaches?). If they are different, I wonder if that could hurt performance on harder tasks and if that could be fixed somehow (akin to weight symmetry in Akrout et al.?)\n\nIs CorInfoMax here similar to the Information Bottleneck ideas?\n\nThe abstract says\n> The backpropagation algorithm... it remains an open question  whether the brain employs supervised learning mechanisms akin to it\n\nBackprop doesn’t imply supervised learning (see VAEs, self-supervised methods and so on). The overall claim is fine, but it shouldn't be about supervised learning.\n\nEq. 27: should the second $e^{k+1}$ be $e^k$?\n\n**Overall**, it is an interesting contribution but the experiments are very small-scale and the proposed approach is not compared to the main competitor, backpropagation.\n\n----------\n**Post-rebuttal**: feedback alignment-level performance is a limitation of this work, but the principled approach to derive multi-compartment models and additional evaluations done during the rebuttal justify an increase of the score from 5 to 6.\n", "limitations": "The limitations and potential impacts have been addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The (approximation to the) CorInfoMax objective produces a tractable model of a neuron with several compartments. This resonates with previous ideas of credit assignment with apical dendrites, and (I guess) generates experimentally testable predictions due to the specific interactions between compartments and weights. \n\nThe approach to weight symmetry is interesting and might implicitly lead to weight symmetry (although see weaknesses). \n\nOverall, this is a novel idea, even though it is very related to previous works that use apical dendrites/predicting coding/etc. as a mechanism for credit assignment. ", "weaknesses": "The experiments in Tab. 1 have multiple problems. There's no comparison to backprop and no explanation of used architectures in the main text. Presumably the architectures were pretty small, given poor CIFAR10 performance.\n\nRelated, all experiments show feedback alignment-level performance (i.e. good on MNIST, OK on CIFAR10 for a small network that reaches about 50% accuracy). Thus, we can’t draw any conclusions about the effectiveness of this approach without considering at least larger networks and maybe harder datasets (as feedback alignment doesn't scale beyond those cases). The minimum aim would be to train a standard ResNet18 on CIFAR10 with backprop (should be around 90% accuracy), and compare it to all algorithms in Tab. 1. \n\nThe authors also missed an important previous work -- Deep Learning without Weight Transport by Akrout et al. (2019). That paper proposes a simple mechanism for the weight transport problem that is a bit different from the one here, but it is still worth discussing in the context of backprop approximations/alternatives. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful feedback. Due to strict length constraints, we had to omit some details in our responses. We eagerly look forward to providing further clarifications and answering any additional questions during the discussion period.\n> Strengths:\n\nThank you for your positive comments.\n>..might implicitly lead to weight symmetry..\n\nOur theoretical discussion in Appendix B.2, supported by experimental angle measurements between feedback and transpose feedback weights in Appendices B.3, E.4.3, and E.5.4, indicates that CorInfoMax networks do not lead to implicit weight symmetry.\n> Weaknesses:\n>..no comparison to backprop..\n\nIn the revised article, we've updated Table 1 with backpropagation and feedback alignment results, available in the rebuttal PDF.\n>.. no explanation of used architectures..\n\nThe revision incorporates architecture information into the main text. Our initial submission's appendix detailed hyperparameters and network structures for each experiment. We used 2- or 3-layer fully connected networks with hidden sizes of $500$ for MNIST and $1000$ for CIFAR10 in 2-layer networks, and $500$-$500$ for MNIST and $1000$-$500$ for CIFAR10 in 3-layer networks.\n>...to train a standard ResNet18 ...\n\nOur key objective is to propose a normative learning rule leading to segregated pyramidal neuron models, addressing the weight transport problem. We focused on fully connected neural networks to establish our theoretical foundations. In our new experiments, we've achieved similar performance to biologically plausible benchmarks and BP-trained fully connected networks. We deliberately excluded CNNs, including ResNets, due to their weight-sharing feature that doesn't align with locally connected neuronal models in the brain. \n>... also missed an important previous work -- Deep Learning without Weight Transport by Akrout...\n\nThanks a lot for pointing this relevant reference. We inserted the following change to Section 1.1.2: *\"For example, the feedback\nalignment approach, which fixes randomly initialized feedback weights and adapts feedforward weights, was offered as a plausible solution [17]. Later Akrout et.al. [18 ] proposed its extension by updating feedback weights towards to the transpose of the feedforward weights\"*\n>Questions:\n>..are the feedback weights different from the feedforward at the end of training?..  If they are different, I wonder if that could hurt performance..\n\nWe appreciate the opportunity to clarify our model's distinctiveness. As discussed in Appendices B.2, B.3, E.4.3, and E.5.4, our model's feedback weights are not transposes of the feedforward weights, unlike conventional backpropagation networks. As discussed in Appendix B.2 of our initial submission, we can write the forward and backward predictor weights as\n$$\\mathbf{W}\\_{ff,\\*}^{(k)}=\\mathbf{R}\\_{\\mathbf{r}^{(k+1)}\\mathbf{r}^{(k)}}(\\mathbf{R}\\_{\\mathbf{r}^{(k)}}+\\epsilon_k \\mathbf{I})^{-1},$$ \n$$\\mathbf{W}\\_{fb,\\*}^{(k)}=\\mathbf{R}\\_{\\mathbf{r}^{(k)}\\mathbf{r}^{(k+1)}}(\\mathbf{R}\\_{\\mathbf{r}^{(k+1)}}+\\epsilon_k \\mathbf{I})^{-1}$$\nInspecting these, they do not just involve $\\mathbf{R}\\_{\\mathbf{r}^{(k+1)}\\mathbf{r}^{(k)}}$ and $\\mathbf{R}\\_{\\mathbf{r}^{(k)}\\mathbf{r}^{(k+1)}}$ which are transposes of each other, but also the inverse autocorrelation matrices $(\\mathbf{R}\\_{\\mathbf{r}\\^{(k)}}+\\epsilon_k \\mathbf{I})^{-1}$ and $(\\mathbf{R}_{\\mathbf{r}^{(k+1)}}+\\epsilon_k \\mathbf{I})^{-1}$. Consequently, the condition $\\mathbf{W}\\_{ff}^{(k)}={\\mathbf{W}\\_{fb}^{(k)}}^T$ does not generally hold true. Symmetry might be anticipated for diagonal autocorrelation matrices. \n\nIn the feedforward network based standard backpropagation setting, the output of the feedback network does not directly influence the feedforward network's output, instead it generates credit signals for updating the feedforward weights. For output mean square error minimization, the feedback weights should be the transposes of the feedforward weights.\n\nHowever, CorInfoMax networks operate differently. Being recurrent network with feedback, these networks' dynamics—and thus their intermediate and output signals—are directly influenced by the feedback weights. Our use of equilibrium propagation-based learning ensures that the weights adapt to minimize the mean square error loss function, guided by the CorInfoMax objective. Consequently, there's no requirement for feedback weights to mirror the feedforward weights.\n> Is CorInfoMax here similar to the Information Bottleneck ideas?\n\nWhile both CorInfoMax and the Information Bottleneck principle derive from information theory, their relationship isn't straightforward. Traditionally, the Information Bottleneck method aims to maximize the Shannon Mutual Information (SMI) between a hidden vector and the output label, whilst simultaneously minimizing its SMI with the input. This dual goal ensures the relevance of the hidden layer to the output while promoting compression.\n\nOn the other hand, the CorInfoMax framework is designed to maximize the correlative information flow across the input, hidden layers, and output in a bidirectional fashion. CorInfoMax achieves potential compression by adopting specific domain sets, such as polytopes, for the hidden and output layers. Consequently, this leads to piecewise linear activation functions and lateral inhibition neurons at these layers.\n> ..Backprop doesn’t imply supervised learning...\n\nWe agree. We focused on supervised learning, primarily due to its notational convenience and alignment with the traditional form of backpropagation. However, it's crucial to note that our framework could be feasibly extended to other unsupervised and self-supervised paradigms.\n> ... should the second $\\mathbf{e}^{k+1}$ be $\\mathbf{e}^k$? \n\nThanks. We corrected this in the revision.\n> ...  the proposed approach is not compared to the main competitor, backpropagation.\n\nWe included new experiments with BP. Please see rebuttal pdf for the updated Table 1."}, {"review_id": "0DHhKYPAyW", "reviewer": "Reviewer_K4kq", "summary": "The authors introduce a biologically plausible training paradigm for a deep neural network that sidesteps the weight transport problem while achieving competitive results. Their approach is normative, in that both the network's architecture as well as its learning rules can be derived from an information maximization approach. The asymmetry between forward and backward weights is achieved by leveraging two different formulations of the inter-layer correlative information.", "questions": "Questions\n* for me, phrasing the method in terms of using two *equivalent* forms of the correlative mutual information is misleading: if they are equivalent, how could they lead to the desired asymmetry between feed-forward and feed-back weights? I may be wrong, but I believe that the resolution lies in the Taylor expansion around line 178 -- it's not the differing exact expressions that lead to the asymmetry, but the different *approximations*\n* the definition of $\\mathbf M$ on line 212 is a bit confusing. Should the factor of 2 apply to both terms? I would have thought that the leak term in the expression for $\\mathbf M$ is supposed to counteract the leak term in eq. (15) as long as $\\mathbf u$ is inside the feasible domain\n* also regarding $\\mathbf M$: since $\\epsilon$ is small, my understanding is that $\\mathbf M$ is also small; in this case, I would suggest leaving $\\epsilon$ out of the definition for $\\mathbf M$, to make the scale of each term more apparent; e.g., in eq. (18), I would imagine $\\mathbf W_{fb}$ has the leading contribution to $\\mathbf v_A$, while the $\\mathbf M$ term is a sub-leading correction\n* were hyperparameters optimized for each task for all of the algorithms in Table 1?\n\nMinor comments:\n* in eq. (12), the notation $\\hat J_k$ is used but it had never been introduced before. Please define $\\hat J_k$ first before using\n* the update rule for the lateral coefficients (lines 280-281) should be written in terms of the biologically motivated parameters ($\\mathbf M$, or even better, $\\mathbf D$ and $\\mathbf O$) instead of $\\mathbf B$.\n* in the numerical experiments section, please include at least basic details about the networks that are used (e.g., number of layers)\n* below eq. (A.12), the clipping operation $\\sigma_+$ is invoked out of nowhere. I think the point is to show how this operation can be justified as a way of enforcing the KKT conditions; this should be made clearer\n* the plots in Appendix Figures 7–8, 12–14 are very sparse and it's not clear what information we are to glean from them; I suggest combining some of these, e.g., compare the accuracy attained by different network architectures and / or activation functions\n* in the studies from Appendix F, the range over which the hyperparameters are varied seems too small because the variation in accuracy is very low, barely beyond the variability over different runs; I suggest using larger variations to show the trend", "limitations": "The authors have adequately discussed limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "*Originality:*\nThe work provides a novel approach for deriving biologically plausible strategies for learning in deep neural networks.\n\n*Quality:*\nThe paper contains a significant amount of work to support its findings. Importantly, both theory and computation are used in tandem.\n\n*Clarity:*\nThe presentation is mostly clear, but some significant explanations are missing or too sparse. See below. I also want to praise the authors for including the code that they used with the submission (something that I believe should be true for all papers, but sadly is not).\n\n*Significance:*\nThe work is significant for neuroscience because the learning algorithms used by the brain are not yet understood. Having a good grasp over the range of possible mechanisms that biology could have used to train natural neural networks is essential to allow experimentalists to probe what choice(s) is (are) actually used. The work is also of potential significance for machine learning, since the algorithms used by the brain might provide advantages over the gradient descent with backpropagation methods used to train artificial neural networks.\n", "weaknesses": "1. The correlative mutual information metric requires a bit more discussion. The regularization coefficients $\\epsilon_k$ appear in eq. (2) but are not discussed at all until much later, and even in the derivation in Appendix A, the need for this regularization is not explained. On first guess, the need for $\\epsilon_k \\ne 0$ is due to having a low rank covariance matrix $\\mathbf R_{\\mathbf r^{(k+1)}}$. However, this seems inconsistent with the importance of these coefficients in the network dynamics and learning rules. This requires a more detailed discussion in the main text, and especially in the Appendix. (If space is an issue, I suggest removing most of lines 170-174, which are almost identical to 149-153; it can simply be stated that the sample covariance matrices from eq. (6) need to be used instead of their exact counterparts to get online training rules.)\n2. Related to the regularization coefficients, I am a bit perplexed by eqns. (10), (11). The Taylor expansion in these equations is performed around the identity, but that makes the expansion parameter be $1 / \\epsilon$. Since $\\epsilon$ is small, $1 / \\epsilon$ should be big, making it hard to justify ignoring subsequent terms in the Taylor expansion. This needs to be explained in detail.\n3. The jump to the dynamics equations (15)–(17) is too abrupt. Either an explanation should be provided or a reference to a relevant Appendix section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed review and constructive comments. Due to the strict length constraint some details we prepared were needed to be removed. We will be happy to provide more details and answer your potential questions during the discussion period.\n>Strengths:\n\nWe sincerely value your recognition of our work's novelty and significance\n> Weaknesses:\n>..Discussion on CMI metric, and regularization parameter $\\epsilon_k$..\n\nThanks a lot for this constructive suggestion. Please see items 2 - 3 in our global rebuttal response.\n>..Taylor exp. in (10-11) is performed around the identity..\n\nOur revised article includes an appendix section detailing the linearization based on the truncated Taylor series. Briefly, the linearization around $\\mathbf{A}$ with perturbation $\\mathbf{\\Delta}$ can be expressed as\n$\\log\\det(\\mathbf{A}+\\mathbf{\\Delta})\\approx \\log\\det(\\mathbf{A})+\\text{Tr}(\\mathbf{A}^{-1}\\mathbf{\\Delta}).$\nFor the correlative entropy of the prediction error,  $\\log\\det(\\epsilon \\mathbf{I}+\\mathbf{R}\\_\\mathbf{e})$, we assume $\\epsilon \\mathbf{I} \\succ \\mathbf{R}\\_\\mathbf{e}$ and  linearize around $\\mathbf{A}=\\epsilon \\mathbf{I}$ with perturbation $\\mathbf{\\Delta}=\\mathbf{R}\\_\\mathbf{e}$, yielding $\\log\\det(\\epsilon \\mathbf{I}+\\mathbf{R}\\_\\mathbf{e})\\approx \\log\\det(\\epsilon \\mathbf{I})+\\epsilon^{-1}\\text{Tr}({\\mathbf{R}\\_\\mathbf{e}})$.\n In summary, the perturbation term is not $\\epsilon \\mathbf{I}$ but ${\\mathbf{R}\\_\\mathbf{e}}$. The assumption follows from the discussion on the impact of $\\epsilon$, where maximizing the CMI is achieved by pushing the eigenvalues of ${\\mathbf{R}_\\mathbf{e}}$ below $\\epsilon$, for *reasonable* values of $\\epsilon$. For our nominal choice of $\\epsilon=0.15$ in our experiments, it is indeed the case.\n>..jump to the dynamics equations (15-17) is too abrupt..\n\nThanks again for this suggestion. We  include a new appendix section on the more detailed derivation of (15-17).\n>Questions:\n>phrasing the method in terms of using two equivalent forms of the CMI is misleading: .. how could they lead to the desired asymmetry between feed-forward and feed-back weights? ..it's not the differing exact expressions that lead to the asymmetry, but the different approximations?\n\nBriefly, asymmetry is **not** due to Taylor series based approximation: Equations (7) and (8) indeed represent two equivalent alternatives for  Correlative Mutual Information (CMI), but individual components vary. Specifically,   the first terms of (7) and (8) represent the Correlative Entropy (CE) of layer $k+1$ and layer $k$ activations, while the second terms are the CEs of forward and backward prediction errors. These are not necessarily equivalent,  leading to inherently unequal forward and backward error entropies and corresponding weight matrices. Taylor series based approximation is just for the linearization of the forward and backward prediction entropies. As discussed in Appendix B.2, we can write the forward and backward predictor weights as\n$\\mathbf{W}\\_{ff,\\*}^{(k)}=\\mathbf{R}\\_{\\mathbf{r}^{(k+1)}\\mathbf{r}^{(k)}}(\\mathbf{R}\\_{\\mathbf{r}^{(k)}}+\\epsilon_k \\mathbf{I})^{-1},$ \n$\\mathbf{W}\\_{fb,\\*}^{(k)}=\\mathbf{R}\\_{\\mathbf{r}^{(k)}\\mathbf{r}^{(k+1)}}(\\mathbf{R}\\_{\\mathbf{r}^{(k+1)}}+\\epsilon_k \\mathbf{I})^{-1}$\n $(\\mathbf{R}\\_{\\mathbf{r}^{(k)}}+\\epsilon_k \\mathbf{I})^{-1}$ and $(\\mathbf{R}\\_{\\mathbf{r}^{(k+1)}}+\\epsilon_k \\mathbf{I})^{-1}$. Consequently, the condition $\\mathbf{W}\\_{ff}^{(k)}={\\mathbf{W}\\_{fb}^{(k)}}^T$ does not generally hold true. Symmetry might be anticipated in very specific scenarios - such as diagonal  autocorrelation matrices. This analysis only considers the mutual information maximization component of the objective, yet it offers insight into the expected asymmetry of the forward and backward weights. \n>..the definition of $\\mathbf{M}$.. the factor of 2 apply to both terms?\n\nThanks for pointing the typo. We fixed as $\\mathbf{M}^{(k)}[t] = \\epsilon_k(2 \\gamma\\mathbf{B}^{(k)}[t] + g_{\\text{lk}} \\mathbf{I}$)\n>... $\\mathbf{M}$ is also small ... I would suggest leaving $\\mathbf{\\epsilon}$ out of the definition...\n\nThanks. The goal was to obtain compact representation. We will evaluate your suggestion for the revision.\n>were hyperparameters optimized for each task for all of the algorithms in Table 1\n\nIndeed, we put considerable effort into optimizing the hyperparameters for most of the tasks and all the algorithms in Table 1, using grid search. Our shared code explicitly include the grid search parameters associated with each algorithm. Additionally, our Python notebooks under \"AnalyzeSimulations\" present the train and test results in a comprehensive table, enabling easy comparison between various settings.\n>Minor Comments:\n>in eq. (12), ..$\\mathbf{\\hat{J}_k}$.. had never been introduced..\n\nThanks. In the revision,  we explicity include ${\\hat{J}}\\_k(\\mathbf{r}^{(k)}) = \\overset{\\rightarrow}{\\hat{I}^{(\\epsilon\\_{k-1})}}(\\mathbf{r}^{(k - 1)},\\mathbf{r}^{(k)})[t]+\\overset{\\leftarrow}{\\hat{I}^{(\\epsilon\\_k)}}(\\mathbf{r}^{(k)},\\mathbf{r}^{(k+1)})[t]$ for $k=1, \\ldots, P$ and $\\hat{J}\\_P(\\mathbf{r}^{(P)})[t]=\\overset{\\rightarrow}{\\hat{I}^{(\\epsilon\\_{P-1})}}(\\mathbf{r}^{(P - 1)},\\mathbf{r}^{(P)})[t]-\\frac{\\beta}{2}\\|\\mathbf{r}^{(P)}[t]-\\mathbf{y}\\_T[t]\\|_2^2$.\n>the update rule for the lateral coefficients ... in terms of ... ($\\mathbf{M}$, or ..., $\\mathbf{D}$ and $\\mathbf{O}$)\n\nWe'll revise the update equations as suggested (see Alg. 1 in global PDF).\n>... experiments section...include...details about the networks..\n\nThanks. We provide them in the appendix but we will include in the main text of the revision.\n>below (A.12)...$\\sigma_+$ is invoked out of nowhere ..\n\nWe reworded  this sentence. \n>... Figures 7–8, 12–14 are very sparse...suggest combining...\n\nFollowing your advice, we merged 7-9 and 12-14.\n>Appendix F...hyperparameters...suggest using larger variations to show the trend...\n\nRevised manuscript will include an extended ablation study section in appendix."}, {"review_id": "UBDa6hioS8", "reviewer": "Reviewer_t54y", "summary": "The authors present a novel strategy for learning in neural networks. In particular, the authors derive update rules for neurons/synapses which maximises the correlative information between layer activations. This strategy avoids the weight transfer problem, and naturally gives rise to a biologically emulating architecture of multi-compartment pyramidal neurons with lateral inhibition. ", "questions": "- The segway at line 34 was unclear to me\n- I found section 2.1 confusing to read because the equation which derives the activity r^k of a given layer is not expressed. Is this intentional to keep it general? It's confusing to understand whether the activities between the layers have a relationship at all at this point. \n- line 140: the inequality used to express the hypercube is not well defined for vectors\n- The CMI metric (equation 2) is a complicated equation with determinants and auto/cross-correlation matrices. I would have liked an intuitve \n (perhaps geometric) description of this measure\n- line 150: R_{r^k r^{k+1}} hasn't appeared yet but is already being described\n- in section 2.3.1 the variable s is introduced without any explanation as to what it represents. Is it time? i.e. u[t, s] is the t dataset example at time s? \n- line 231: I didn't understand the decomposition of M. Where does D come from and why does it mirror autapses? Is it the identity matrix in the expression of M? WHere does the negative O part (interneuron) come from?\n- equations 26,27: sorry if this is a naive question, but why can't the weights just be updated directly using equations 23,25?", "limitations": "I would recommend a limitations section (or at least more discussion). For example, I would be interested to know if the sensitivity of the model to hyperparameter choices is high, or whether there is a strict need for the feedback matrix at the last layer to be the identity.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- the authors present what seems a mathematically sound and creative strategy for credit assignment. Without extensive knowledge in this area, the derivation of update rules seems original and of good quality\n- The resulting likeness to a multi-compartment model with lateral inhibition is interesting\n- the text is generally well written (though the presentation itself is dense, see below)", "weaknesses": "- in general I found the paper very dense - I personally think 27 equations is too many for a main text. I appreciate that the main contribution of this paper is analytical, but the think the authors would do well to sacrifice some of the less key equations (move to SM) to make space for additional intepretation/experiments\n- As stated above, I would have liked to have seen more intepretation and experiments with respect to the model. For example, what predictions does the model make in terms of the balance of bottom-up/top-down signals? How does this change over learning? How does it compare to biology? Same for interneurons\n- The actual performance of the model does not seem too impressive, at least compared to standard backprop (e.g. on the CIFAR-10 dataset). Moreover, given that a key property of the model is to avoid the weight symmetry issue, I would think it sensible to compare the model to backprop with random feedback weights (feedback alignment).\n- I think the authors coud make it more explicit what are the differences between their model and the model in Golkar et al. 2022. In particular, explicitly highlighting the similar and new terms when presenting the mathematical formulation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly value your comprehensive review and constructive suggestions. While length constraints have necessitated brevity in our response, we anticipate the opportunity to discuss further details and address any outstanding queries during the discussion phase\n\n>..paper..dense..less key eq.s (move to SM)..\n\nThanks. We followed your advise. Please see items 1 to 3 in global rebuttal response (GRR).\n\n>... more intepretation and experiments... what predictions does the model make..the balance of bottom-up/top-down signals?\n\nAs noted in GRR-item 4, we've included new experiments.  As for the interpretations you've quoted, we believe our info. theory-based framework can provide insightful contributions and  we are currently focused on development. Although the balance of bottom-up and top-down signals presents an interesting research direction, we haven't researched this area yet. However, our principled approach offer potential insights such as the role of lateral connections in embedding space expansion and avoiding degeneracy, feedback and feedforward connections for prediction to reduce redundancy, and activation functions/interneurons to shape feature space and compress. We will elaborate more on these in the revision by benefitting the extra page.\n>..performance..not..too impressive..compared to standard BP..compare..to BP with random feedback..\n\nFollowing your suggestion, we performed additional experiments for comparisons with BP and feedback alignment for fully connected architecture. Updated Table 1 is available in Rebuttal pdf, which shows that CorInfoMax has on par performance with these benchmarks.\n>.. more explicit..differences between Golkar et al. ..highlighting...the mathematical formulation\n\nWe can provide the following comparison between the CorInfoMax and the constrained predictive coding (C-PC) framework in Golkar et al.:\n\n* C-PC enhances existing maximum likelihood (ML) based predictive coding frameworks by integrating secondary forward prediction terms into the ML formulation. The minimization of negative log-likelihood can be expressed as:\n\\begin{align}\n\\min\\_{\\mathbf{Z},\\mathbf{W}\\_a,\\mathbf{W}\\_b} \\hat{L}=\\frac{1}{2}\\sum_{l=1}^{n-1}\\left[\\frac{ \\|\\mathbf{Z}^{(l)}-\\mathbf{W}^{(l-1)}_b\\mathbf{Z}^{(l-1)}\\|_F^2}{2{\\sigma^{(l)}}^2}+\\frac{ \\|\\mathbf{Z}^{(l+1)}-\\mathbf{W}^{(l)}_a\\mathbf{Z}^{(l)}\\|_F^2}{2{\\sigma^{(l+1)}}^2}\\right].\n\\end{align}\nNote that both terms in the summation are two separate forward prediction error terms.\n\n* In CorInfoMax, we propose maximization of correlative mutual information between sequential branches, where both forward and backward prediction matrices emerge from two alternative but equivalent forms of the CMI,\n* C-PC approach makes use of a whitening constraint on layer activations, which is utilized to convert forward prediction matrix $\\mathbf{W}_a$ to a feedback matrix $\\mathbf{W}_a^T$,\n* In CorInfoMax framework,  there is no whitening but a set membership constraint on layer activation vectors.\n* In C-PC lateral weights are based on Lagrangians of the covariance constraints,\n* In CorInfoMax lateral weights are inverse of the activation correlation matrix to maximize correlative entropy of activations,\n* The updates for feedforward and feedback matrices are different for both approaches (forward and backward prediction errors are used in the CorInfoMax).\n\n>Questions:\n>segway at line 34..unclear.. \n\nThe first paragraph addresses two main critiques concerning bio. plausibility: weight transport and simple neuron models. The subsequent paragraph delves into the weight symmetry issue, while the third one explores neuron models. We will ensure a more seamless transition between these.\n\n>sec. 2.1 confusing..the eq. which derives the activity $\\mathbf{r}^k$..is not expressed. Is this intentional?\n\nIndeed, this is intentional. Equations reflecting activity, network structure, dynamics, and learning updates are not predetermined but they emerge from correlative information maximization with activation domain constraints. \n\n>line 140: the ineq..is not well defined\n\nApologies for notation ambiguity. The ineq. $\\mathbf{0} \\leq \\mathbf{r} \\leq \\mathbf{1}$ denotes elementwise comparison. We can change $\\leq$ to $\\preccurlyeq$.\n\n>The CMI metric (2) is a complicated..an intuitve (perhaps geometric) descrip. of this measure\n\nThanks for the suggestion. We included the following in Sec. 2.2: *\"If we interpret the maximization of CMI in (2): the first term on the right side of (2) encourages the spread of $\\mathbf{r}^{(k+1)}$ in its presumed domain \n$\\mathcal{P}^{(k+1)}$, while the second term incites the minimization of redundancy in $\\mathbf{r}^{(k+1)}$  beyond its component predictable from $\\mathbf{r}^{(k)}$.\"*\n\n>line 150: $\\mathbf{R}_{\\mathbf{r}^k \\mathbf{r}^{k+1}}$ hasn't appeared yet \n\nThanks. Removed. See GRR item 2.i.\n\n>in sec. 2.3.1..s is introduced without any explanation..\n\nSection 2.1 defines $t$ as the discrete data index, while $s$ is the continous time index for the network dynamics. We clarify this in the new appendix for network dynamics.\n\n>... the decomposition of M. Where does D come from ...\n\nWe clarify it in the revision. Briefly, we define $\\mathbf{D}^{(k)}$ as $\\mathbf{D}^{(k)} = \\text{diag}(\\mathbf{M}^{(k)})$, i.e., a diagonal matrix containing the diagonal elements of $\\mathbf{M}^{(k)}$. Then we define $\\mathbf{O}^{(k)} = \\mathbf{D}^{(k)} - \\mathbf{M}^{(k)}$. Therefore, we can express $\\mathbf{M}^{(k)}$ as $\\mathbf{M}^{(k)} = \\mathbf{D}^{(k)} - \\mathbf{O}^{(k)}$.\n\n>..(26-27):..why can't the weights just be updated..using..23,25?\n\nEquilibrium Propagation updates in 26,27 target minimization of MSE error, whereas 23,25 are gradients for  the CMI based energy function determining system dynamics. The use of 23,25 still provides some accuracy but significantly below the EP updates.\n\n>Limitations: \n>I would recommend a limitations section ...\n\nThanks, we include a limitations section following your suggestions. See GRR item 5."}, {"review_id": "VPlgbpu51V", "reviewer": "Reviewer_wEH2", "summary": "This paper seeks to present a biologically plausible learning approach for supervised learning in deep neural networks. Unlike backpropagation, the approach does not require symmetric weights in the forward and backward directions. The approach relies on an information theoretic approach which seeks to maximize mutual information between layers in the forward and backward direction. The approach is demonstrated on simple data sets (e.g. MNIST) as well as on 3-compartment models of pyramidal neurons.", "questions": "It is not clear what you do at the top layer with the targets and the error.  Are you clamping the output layer to the targets and then applying your backward pass?\n\nThe term \"disputed\" in the abstract is too weak. I think there is a strong general consensus that plain backpropagation is not biologically plausible. \n\nline 42: \"maximize the linear dependence\"--why linear, since the networks are typically non-linear? furthermore it is easy to maximize linear dependence by making the signals identical (i.e. identical activities in two neighboring layers of the same size, which is NOT interesting.\n\nline 59: \"predictors\"  of what?\n\nFigure 1 is not easy to understand at this stage of the paper. In fact, the use of compartmental models at the beginning of the paper is confusing. Maybe these should be moved entirely toward the end of the paper or in an appendix to improve readability.\n\nTo the best of our knowledge, the result by Liao et al is not reliable --it has not been confirmed systematically.\n\nThe term polytopic is not very common and should be defined. Is this assumption needed in the case of artificial neural networks?\n\nline 148: why do you mention that the objective is stochastic, and stochastic in which sense? Also, to be clear, you should specify whether you are trying to maximize or minimize the objective.  \n\nIs there some Gaussian assumption behind equation 2 (if so it should be stated explicitly). Also, any connection or divergence from better-known concepts, such as mutual information, should be clarified.\n\nEquation 2 and what follows are particularly unclear and hard to follow.\n\n153:  \"problem\" Which problem exactly?\n\npage 6: the learning rule is not clear. Can you write it in the standard form of  \"\\delta w_{ij}=learning rate x update urle....$?", "limitations": "See some of the remarks above. There is no discussion of the limitations of the proposed approach.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The manuscript addresses an important problem--essentially how can supervised learning be implemented in biological neural networks. It proposes a solution to the well-known weight symmetry/transport problem. And it seeks to do so in a principled way using information theoretic notions", "weaknesses": "Unfortunately, the paper is poorly written, with heavy notations and equations which often obscure the approach rather than clarifying it.\n\nThere is no clear expression for the learning algorithm. It is hard to see that the learning algorithm is local both in space and time, which is a major requirement in a biologically plausible network.\n\nFor the experiments, the authors report the test accuracy. However other metrics would also be interesting, for instance, the degree of symmetry between the final weights in both directions. \n\nIt seems that the approach still requires propagating error information over long distances (across many layers) which may also be problematic from a biological point of view.\n\nSupervised learning is not particularly biologically plausible.  This point should be addressed, as a minimum using self-supervised learning in combination with the proposed approach.\n\nThe authors should mention more clearly that the weight transport problem is completely solved by random backpropagation or feedback alignment. Thus the advantages of their approach, if any, should be contrasted with feedback alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed review and  useful feedback. We  had to truncate some of our responses to adhere to the length constraints.  We are very eager to provide more details for any questions you might have during the discussion period.\n>..poorly written..heavy notations..\n\nWe acknowledge your feedback concerning the accessibility. In the revision, we refined the presentation in Section 2 by relocating certain equations to appendix and providing more comprehensive explanations for enhanced clarity. For a detailed outline of these changes, please refer to items 1 to 3 in our global rebuttal response (GRR).\n>..no clear expression for learning..hard to see the algorithm is local.. \n\nIn initial submission, we presented the learning rule (see Eq. 26-27) with limited details due to space constraints. To rectify this, we've enriched our revision with an appendix section, \"Learning Dynamics\", with succinct pseudocode for our algorithm and additional elaboration, ensuring the locality constraints are clearly met. Refer to Alg. 1 in our global rebuttal PDF for more details.\n>..other metrics would be interesting, e.g., the degree of symmetry..\n\nWe indeed evaluated the angle between forward and backward weights using the cosine angle metric (Eq. A.7), detailed in our appendix. Refer to Fig. 2 (App. B.3) for the angle's evolution in our 2-layer experiments, starting with $90^\\circ$ converging to $\\sim70^\\circ$. Figs 10&15 illustrate the angle patterns in 3-layer case and sparse networks, highlighting the inherent asymmetry for our framework.\n>..prop. error information over long distances..\n\nThe CorInfoMax networks  solely include feedback projections from the next layer in the hierarchy, aligning with known biological connectivity patterns. They lack a separate error prop. mechanism or direct long-range top-down projections\n>..Supervised learning is not particularly bio-plausible..\n\nWe recognize the essential role of unsupervised/self-supervised learning in natural learning processes. Our main contribution lies in the dynamics and structure of info. propagation, which can be extended to unsupervised objectives. This point will be clarified in the revision as per your valuable suggestion.\n>..weight transport problem is completely solved by random backprop.. \n\nWe agree that random BP is one plausible solution. We revised Sec. 1.1.2 to include \"For example, the feedback alignment approach, which fixes randomly initialized feedback weights  and adapts feedforward weights, was offered as a plausible solution [17]\". CorInfoMax is offered as an info theory based principled alternative hypothesis where networks of segregated neurons with recurrent and asymmetric feedback connections governed by local learning rules naturally emerge. Such a normative framework is useful in obtaining potential insights such as the role of lateral connections in embedding space expansion and avoiding degeneracy, feedback and feedforward connections for prediction to reduce redundancy, and activation functions/interneurons to shape feature space and compress. We will elaborate more on these in the revision by benefitting the extra page.\n>Questions\n>..not clear..at the top layer\n\nWe utilize weak, not full, clamping based on the objective in (9a). As shown in Eq. (21) for the output layer, the error between the network output and labels influences network dynamics via the gain $\\beta$.\n>\"disputed\" is too weak\n\nWe agree. We will consider replacing the word \"disputed\" in the revised article.\n>\"maximize linear dependence\" why linear..identical activities..\n\nThis is an important point to be clarified: typical network models use linear segments between layers (modelling synaptic integration), succeeded by nonlinear activations. In our framework, these linear segments in both directions emerge via correlation maximization. Additionally, set membership constraints, like the $\\ell_1$-norm ball for sparsity, bring in nonlinearity. Thus, **linear mappings followed by nonlinear activtions** emerge from **CMI maximization under  domain set constraints for layers**.\n>\"predictors\" of what?\n\nTo clarify, in the revised article we write \"predictors of layer activation signals\".\n>Fig. 1 is not easy to understand at this stage..\n\nWe value your suggestion. Our aim is to maintain Fig.1 early on to offer a preview and motivate the discussion, if feasible.\n>..result by Liao..\n\nWe will look into it more closely, thanks.\n>The term polytopic is not common..\n\nPolytopes, as compact intersections of half-spaces, allow flexible characterization of bounded layer activations. The choice of a polytope influences feature combinations like sparsity, antisparsity, and nonnegativity. Imposing these constraints leads to piecewise activations like ReLU and clipping functions, and introduces interneurons for sparsity. We will be happy to provide a brief summary as an appendix in the revision.\n>..stochastic in which sense?..specify..trying to maximize of minimize..\n\nFirst, we define the obj. function in ensemble average form using expectations. Later, we provide its sample average based form. We modified line 148 to specify the objective is \"to be maximized\".\n>Is there some Gaussian assumption..\n\nWe make no Gaussian assumption. The correlative mutual information (CMI) is a second-order statistics-based measure, independent of the probability density functions, and it assesses the correlation level between its arguments. Please refer to Appendix A for an introduction to correlative entropy and CMI, which we expand in the revision.\n>..particularly unclear and hard to follow..\n\nWe will modify Section 2 for better presentation and accessibility in the revised article.\n>Which problem?\n\nIn the revision, we clarified the term \"problem\" to mean the optimization for finding the optimal linear regularized MMSE predictor, $\\mathbf{W}\\_{ff,\\*}^{(k)}$, given in Eq. (3).\n>..no..limitations..\n\nWe include a limitations section following your suggestion. Please see GRR item 5."}]

Author rebuttal:
We are grateful to all reviewers for their comprehensive evaluations and insightful feedback. This response covers main points and shared concerns.  **Owing to the strict length constraint**, we've made every effort to respond to individual comments and questions here and within each reviewer's rebuttal. **We will be more than happy to provide more details during the discussion phase.**
 
Our article offers a method grounded in information theory for the development of biologically plausible neural networks. Enhancing the clarity and accessibility of the theoretical content has been identified as an area for improvement. In addition, there have been requests for clearer derivations, explanations, and additional experiments. The revision we prepared  addresses these aspects, as highlighted below:

1. Revise Section 2.2 for better readability by *moving some details to the appendix and eliminating some unnecessary expressions*: a) Move linear approximation of correlative entropy (based on Taylor series) in (10)-(11) to a new appendix section, b) transfer CorInfoMax objective function gradient derivation details in (12)-(14) to a new appendix, c) add a new appendix section for the derivation of network dynamics equations (15)-(17), d) eliminate cross correlation matrix and detailed definition of error correlation matrix after (2).

2. Revise Section 2.2 for *better explanations and clarifications*, e.g., 

i. Replace the description after (2), which contains the definition of correlative mutual information, for clarification: "$$\overset{\rightarrow}{{I}^{(\epsilon_k)}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)}) =  \frac{1}{2} \log \det \left(\mathbf{R}_{\mathbf{r}^{(k+1)}} + \epsilon_k \mathbf{I}\right) - \frac{1}{2} \log \det \left(\mathbf{R}\_{\overset{\rightarrow}{\mathbf{e}^{(k+1)}\_\*}} + \epsilon_k \mathbf{I}\right) \quad (2)$$
     is the correlative mutual information between layers $\mathbf{r}^{(k)}$ and $\mathbf{r}^{(k+1)}$, $\mathbf{R}\_{\mathbf{r}\^{(k+1)}}=E(\mathbf{r}^{(k+1)}{\mathbf{r}^{(k+1)}}^T)$ is the autocorrelation matrix corresponding to the layer $\mathbf{r}^{(k+1)}$ activations,  and  
$\mathbf{R}\_{\overset{\rightarrow}{\mathbf{e}^{(k+1)}\_{\*}}}$ corresponds to the error autocorrelation matrix for the best linear regularized minimum MSE predictor of $\mathbf{r}^{(k+1)}$ from $\mathbf{r}^{(k)}$. Therefore, the mutual information objective in (2) makes a referral to the regularized **forward** prediction problem represented by the optimization ...* "

ii. Provide a intuitive description for the maximization of CMI in (2): *"If we interpret the maximization of CMI in (2): the first term on the right side of (2), i.e., the correlative entropy of $(k+1)^{\text{th}}$ layer's activation vector, encourages the spread of $\mathbf{r}^{(k+1)}$ in its presumed domain $\mathcal{P}^{(k+1)}$, while the second term, i.e., the correlative entropy of forward prediction error, incites the minimization of redundancy in $\mathbf{r}^{(k+1)}$  beyond its component predictable from $\mathbf{r}^{(k)}$."*

3. Provide **a new appendix section on the role of $\epsilon$ parameter**. To summarize  a) $\epsilon$ sets a finite lower bound for the correlative entropy, b) $\epsilon$ addresses numerical optimization issues since the derivative of the $\log\det$ function is the inverse of its argument, c) $\epsilon$ acts as a regularizer for the forward and backward prediction problems (see (3) and (5) in the main article), d) $\epsilon^{-1}$  can be viewed as an indicator of the sensitivity of the CMI to the prediction error levels. This last property can be viewed from two perspectives:
 - Inspecting the CMI expression (2) above: $\epsilon_k$ is added to eigenvalues of the correlation matrices, and by definition $\mathbf{R}_{\mathbf{r}^{(k+1)}} \succeq {\mathbf{R}}\_{\overset{\rightarrow}{\mathbf{e}\^{(k+1)}\_*}}$. With $\epsilon_k$ chosen below eigenvalues of $\mathbf{R}\_{\mathbf{r}\^{(k+1)}}$, we can assume $\mathbf{R}\_{\mathbf{r}\^{(k+1)}}+\epsilon_k \mathbf{I}\approx \mathbf{R}\_{\mathbf{r}\^{(k+1)}}$. Thus, the choice of $\epsilon_k$ essentially determine how much we can reduce the correlative prediction error entropy in (2) to maximize the CMI since reducing the eigenvalues of the prediction error correlation matrix below $\epsilon_k$ would not significantly decrease the prediction error entropy. As a result, smaller $\epsilon_k$ implies more emphasis on decreasing prediction error entropy. This is in accordance with how $\epsilon^{-1}$ acts as a conductance parameter channeling prediction errors to output computation. To underline this connection, we will add additional explanation to the discussion at the end of Section 2.3.1: "The inverse of the regularization coefficient $\epsilon_k$ is related to the conductance between soma and dendritic compartments. This is compliant with the interpretation of the $\epsilon^{-1}$ in Appendix A.2 as the sensitivity parameter that determines the contribution of the prediction errors to the CMI."
 - Alternatively, consider the approximation  of (2) with linearized prediction error entropy where $\epsilon_k^{-1}$ appears as the scale of prediction error matrix dependent term:  $$\overset{\rightarrow}{{I}^{(\epsilon_k)}}(\mathbf{r}^{(k)}, \mathbf{r}^{(k+1)}) \approx \frac{1}{2} \log \det \left(\mathbf{R}\_{\mathbf{r}\^{(k+1)}}\right)- \frac{\epsilon_k^{-1}}{2} \text{Tr} \left(\mathbf{R}\_{\overset{\rightarrow}{\mathbf{e}^{(k+1)}\_*}} \right)+const$$

4. Provide **additional numerical experiments** involving comparison with standard backpropagation and feedback alignment algorithms. Table 1 updated with these experiments are provided in the PDF attachment, which confirm that the CorInfoMax has performance on par with available benchmarks.

5. Provide a section on the **limitations** of the proposed framework including hyperparameter sensitivity, contrastive optimization, and training time of our method.

Meta-review: This paper examines the question of "biologically plausible learning", i.e. how can we develop learning algorithms for ANNs that work as well as backpropagation but without some of the key physiological contradictions, such as weight symmetry between the forward and backward pathways. The solution presented in this paper is to use correlative information maximization, and presents two different ways for estimating this quantity, permitting the development of mutli-compartment models that side-step the weight symmetry issue.

The reviewers raised a number of concerns, principally with respect to comparison to other techniques for dealing with weight symmetry and some issues of clarity regarding the derivations and characterizations of the analysis. But, after a fairly extensive back-and-forth, 3/4 reviewers agreed that the paper was sufficiently interesting and makes a worthwhile contribution appropriate for NeurIPS. Altogether, this led to a decision of accept.
Decision: Accept (poster)