A Normative and Biologically Plausible Algorithm for Independent Component Analysis
Authors: Yanis Bahroun
Dmitri Chklovskii
Anirvan M. Sengupta
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Blind source separation
independent component analysis
neural network
local learning rules
biologically plausible
Hebbian learning

Abstract: The brain effortlessly solves blind source separation (BSS) problems, but the algorithm it uses remains elusive. In signal processing, linear BSS problems are often solved by Independent Component Analysis (ICA). To serve as a model of a biological circuit, the ICA neural network (NN) must satisfy at least the following requirements: 1. The algorithm must operate in the online setting where data samples are streamed one at a time, and the NN computes the sources on the fly without storing any significant fraction of the data in memory. 2. The synaptic weight update is local, i.e., it depends only on the biophysical variables present in the vicinity of a synapse. Here, we propose a novel objective function for ICA from which we derive a biologically plausible NN, including both the neural architecture and the synaptic learning rules. Interestingly, our algorithm relies on modulating synaptic plasticity by the total activity of the output neurons. In the brain, this could be accomplished by neuromodulators, extracellular calcium, local field potential, or nitric oxide. 

A Normative and Biologically Plausible Algorithm for Independent Component Analysis
Yanis Bahroun 1.2
Dmitri B. Chklovskii 1.4
Anirvan M. Sengupta 2,3,5
Center for Computational Neuroscience, Flatiron Institute
2 Center for Computational Mathematics, Flatiron Institute
Center for Computational Quantum Physics, Flatiron Institute
Neuroscience Institute, NYU Medical Center
Department of Physics and Astronomy, Rutgers University
org anirvans paw.sesssssicciaaaiiia.. com
Abstract
The brain effortlessly solves blind source separation (BSS) problems, but the algorithm uses remains elusive. In signal processing, linear BS$ problems are often solved by Independent Component Analysis (ICA). To serve as model of a biological circuit, the ICA neural network (NN) must satisfy at least the following requirements: 1. The algorithm must operate in the online setting where data samples are streamed one at time, and the NN computes the sources on the fly without storing any significant fraction of the data in memory. 2. The synaptic weight update is local, i.e., it depends only on the biophysical variables present the vicinity of synapse. Here, propose novel objective function for ICA from which we derive a biologically plausible NN, including both the neural architecture and the synaptic learning rules. Interestingly, synaptic plasticity by the total activity of the output neurons. In the brain, this could be accomplished by neuromodulators, extracellular calcium, local field potential, or nitric oxide.
algorithm relies modulating
1 Introduction
In the brain, visual, auditory, and olfactory systems effortlessly identify latent sources from their mixtures 2 4 In unsupervised learning, such task is known blind source separation (BSS) (5)
BSS is often solved by Independent Component Analysis (ICA) 6 7. which assumes generative model, wherein the observed stimuli are linear combinations of independent sources. ICA algorithms determine the linear transformation back from the observed stimuli into their original sources without knowing how they were mixed in the first place.
Developing a biologically plausible ICA algorithm may provide critical insight into neural compu- tational primitives because ICA may be implemented throughout the brain. In particular, receptive fields of V1 neurons may be the result of performing ICA on natural images [8] 9 Similarly, ICA may account for the receptive fields in the auditory system [10]. Moreover, the neural computational primitives used in the visual and the auditory cortex may similar, as evidenced by anatomical similarity and by developmental experiments where auditory cortex neurons acquire V1-like receptive fields when visual inputs are redirected there []] Therefore, ICA may serve as a computational primitive underlying learning throughout the neocortex.
The majority of existing ICA algorithms (5), for example, information-theoreti ones 12 13 44
15
16 do not meet our biological plausibility requirements. In this work, for the biological plausibility of neural networks (NN), we require that they operate the online (or streaming) setting, namely, the input dataset is streamed one data vector at time, and the corresponding output must be computed
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

without storing any significant fraction of the dataset in memory, ii) the weights of synapses in an NN must be updated using local learning rules, i.e., they depend only on the biophysical variables present in only the two neurons that the synapse connects or extracellular space near the synapse. 20 7 operate online but, when extracting multiple components, rely non-local learning rules, i.e., a synapse needs to "know" about the individual activities of neurons other than the two it connects.
Most existing bio-inspired ICA NNs 17 18 19
More biologically plausible ICA algorithms with local learning rules are limited to sources whose kurtosis deviates from the normal distribution in the same direction, are hand-crafted (ad hoc), and lack good theoretical guarantees 21 22 20]. An alternative to analyzing NNs with ad hoc learning rules is the normative approach. In the normative approach, an optimization problem with known offline solution is used as a starting point to derive online optimization algorithm which maps onto an NN with local learning rules. Such a normative approach led to the development of more biologically plausible ICA networks but only in limited settings of either nonnegative ICA component analysis [2
23 24
or bounded
In this work, we develop a biologically plausible ICA neural network inspired by kurtosis-based ICA methods [27
7 28 Specifically, we take inspiration in the Fourth-Order Blind Identification (FOBI) procedure which separates sources with distinct kurtosis [27]. In this context, distributions are often distinguished depending on their kurtosis relative to a Gaussian distribution, i.e., super- and sub-Gau Gaussian distribution known respectively as leptokurtic ("spiky") and platykurtic ("flat-topped"". Our normative approach is based on a novel similarity-prsserinng objective for ICA with an intuitive geometric interpretation. We reformulate this objective as a min-max optimization problem and solve online by stochastic gradient optimization. We demonstrate that our algorithm performs well on synthetic datasets, audio signals, and natural images.
Our online algorithm maps onto single-layer NN that can separate independent sources without pre-processing. The synaptic weights in our NN are updated using local learning rules, extending more conventional Hebbian learning rules by time-varying modulating factor, which a function of the total output activity. The presence of such a modulating factor suggests a role of the extracellular environment on synaptic plasticity. Modulation of the plasticity rules by overall output activity agrees with several experimental studies that have reported that a third factor, in addition to pre- and post-synaptic activities, can play a crucial role in modulating the outcome of Hebbian plasticity. This extracellular calcium 33 local field
could be accomplished by neuromodulators 129 30
32
potential [3] or nitric oxide 35 36 37.
2 Problem statement and inspiration
The problem of BSS consists of recovering set of unobservable source signals from observed mixtures. When mixing is linear, BSS can be solved by ICA, which decomposes observed random vectors into statistically independent variables.
Mathematically, ICA assumes the following generative model. There are d sources recorded T times forming the columns of S := [s1, ......... ..l....,.77 ST] RxxT whose components st ..,...,5f se are assumed non-Gaussian and independent. Without loss of generality, we assume that each source has zero- mean, unit variance, and finite kurtosis. We also assume that sources have distinct kurtosis as is commonly done in kurtosis-basse ICA methods
(28). The kurtosis of random variable is defined as kurtfe] = R [(v- R[ / (E [(n- - E)) 2 Finally, sources are linearly mixed, i.e., there exists a full rank mixing matrix, A E pdxd producing the d-dimensional mixture, Xy;
As;
W (1...,,7)
(I)
Then the goal of ICA algorithms is to determine linear transformation of the observed signal, WICA E Rdxd such that
Y: = WICAX, Vt (1....,
(2)
recovers unknown sources possibly up to permutation and sign flip.

---

A Three Steps of the FOBI Procedure Illustrated 2D Dataset
(i) Input/mixed data, X
(ii) Whitened data, hg
(iil) Weighted data, 7
(iv) Recovered sources, Yr
source source 2
PCs of
Combined
objective function
X C 1/2
I0
xxPC,
xxx 1/2 C, - Y
B Schematic of the Generative Model and De-mixing by the NN we Derived
Input/sensory units
Feedforward synaptic weights
Two-compartment neurons dentritic (c); somatic (y)
W
S
X
Y₁
Recovered sources
Lateral synaptic weight
S
X
Modulating factor (function the total somatic activity)
S,
%
C Y
Recovered sources
Unknown mixing
Unmixing
Figure 1: A. The three steps of the FOBI procedure illustrated 2D dataset. The purple and pink arrows show the axes of the independent sources 8,. (i) the observed signal, x2, (ii) the whitened data, h, (fii) norm-weighted whitened data, 2,, black arrows represent the principal directions of z, (iv) the recovered sources, Y: are projections h, onto the principal directions of z, These three steps combined into single objective function as indicated by color-coding. B. Schematic of the generative model and de-mixing by the NN we derived. The output layer consists of two- compartment neurons whose synapses obey local learning rules. The dendritic compartments of the output neurons whiten data. The somatic compartments reconstruct the sources by rotating the whitened data. The pale blue rounded rectangle represents modulation of plasticity by output activity.
2.1 Review of the FOBI procedure
FOBI algorithm exploits connection between ICA and Principal Component Analysis (PCA) pointed
38 3
227 40 We reproduce the proof of source recovery by FOBI from 27 40 in
Appendix A.
Description of the procedure. FOBI procedure consists of three steps, Fig. TA. First, the data must be whitened, i.e., all components become decorrelated and of unit variance, Fig. (ii).. The
whitening step can be performed using sample covariance, C, = $ Er x,x, as follows:
Step 1: whiten
h, 1/1/2x1
Data whitening often the first step in many ICA algorithms because recovering the sources after whitening corresponds to finding an orthogonal rotation matrix [7]]. Various ICA methods differ how the rotation is chosen. FOBI, the whitened data, h, scaled by its norm and termed z, Then, the directions of h, and with distinct kurtosis recovered by finding the eigenvectors of

---

the sample covariance matrix + Ei=1 Ziz Fig. Aiii..
T
Step Za: transform 2, ||bel| || h; Step 2b: optimize W. =- arg max
Tr
W
Σ
W
WE T
[=1
WW =Ie
Finally, to recover the sources, we project the whitened data, h. onto the rows of W., Fig. (iviv)
Step 3: project Y: W.
Can a biologically plausible NN implement the FOBI algorithm? The first two steps of FOBI do not present a problem with a biological implementation. For example, Step 2b, essentially PCA, can be solved by stochastic gradient ascent algorithm using Oja's learning rule 41
AW; 77 (u.z, u.u Ws)
(3)
Then, Step 2b can be mapped onto a single-layer network with upstream neurons' activity encoding Ze and the output neurons computing the components of encoded in the weights of feedforward synapses. (3) gives the weight update for the feedforward synapses with the learning rate, > 0. However, according to step 3, the final output of FOBI must be Yt. obtained by multiplying the whitened inputs, h, by W without scaling them by their norm. Such output may be computed by another single-layer network with the same feedforward synaptic weights, W.. but that would require weight-sharing (or weight transport). Alternatively, avoiding weight transport would require non-local update rule for W Thus, both alternatives lead to biologically implausible solutions.
Wa 2z where the elements of W are
2.2 Similarity matching for principal subspace analysis
To find a biologically plausible implementation of FOBI, we follow the approach used previously to derive biologically plausible networks for Principal Subspace Projection (PSP) 42). variant of PCA, and other tasks (43 44 45 46 47
48, 49 50 51 52 53 [54). This approach starts from
reformulating the optimization objective for PSP in the so-called Similarity Matching (SM) form
55
min YER
x x - Y Y
(4)
where X Nyl is the data matrix, Y .y...., Nrl the output matrix, and || || the Frobenius norm. In turn, the objective (4) can be optimized by an online algorithm that maps onto single-layer network of linear neurons whose synapses obey local learning rules 44
Whereas the SM approach leads to biologically plausible NNs for solving cigenproblems, it was still unclear how overcome the weight transport challenge arising in the FOBI algorithm implementation. In the next Section, we address this challenge by introducing a novel SM objective for ICA.
dimilarity-preserrve
objective for ICA
To derive single-layer NN for ICA, which can be trained with local learning rules, we adopt normative approach. We design novel objective function, the solution of which projects the whitened data, h, onto the eigenvectors of the covariance of as specified in the FOBI procedure above using the SM approach. Specifically, we propose the following generalized nonlinearly weighted similarity matching objective using the notations, H [h..... hr] and Z == [zi...., 2r] as defined earlier Sec. 2.1 and illustrated in Fig. IA,
min
H
ZZ
H-YAY
s.t. YYY =
(5)
YEREXT
with 2 = diaggA?
X any diagonal matrix with distinct finite positive entries.
To accomplish all the three steps above in single-layer network, rewrite (5) in terms of the input data X by substituting the expressions for H (Step 1) and Z (Step 2a):
T
min YEReX
x C 1/2
Σ
C
1/2 x
1/2 xx, C 1-1/2X YT
(6)
T
TY
t=1

---

Then the global minima of objective 6 recover the sources as formalized by the following theorem:
Theorem 1. Given that the sources are independent, centered, have unit variance, and distinct
kurtosis (c.f. Sec 2 then the global optimal solution for our objective 6 denoted by Y satisfies
=II
where =\t is diagonal matrix with +1's 's the diagonal, and II is permulation matrix, and is thus a solution to the ICA problem.
Proof. We give the detailed proof of the theorem in Appendix A.
4 Derivation of the algorithm
While our objective (6) can be minimized by taking gradient descent steps with respect to Y. this would not lead to an online algorithm because such computation requires combining data from different time steps. Instead, following [44, we introduce auxiliary matrix variables corresponding to synaptic weights, which store sufficient statistics allowing for the ICA computation using solely instantaneous inputs. Such substitution leads to a min-max optimization problem that is solved by gradient descent/ascent. A corresponding online optimization algorithm using stochastic gradient descent/ascent maps onto an NN with local learning rules.
4.1 Min-max formulation
Here we modify the objective (6) by introducing auxiliary variables, namely W and M, leading to min-max optimization problem. In the following sub-sections, the gradient descent/ascent optimization of the min-max objective will lead to an online algorithm that maps onto an NN where W and M correspond to synaptic weights.
We expand the square in Eq. 6 normalizing by T2, and dropping terms that do not depend on Y yielding:
min T T
Tr TY
s.t. T
YY = Ia
(8)
where, for convenience, we introduce
r, = C=
T Σ 1|
|0-1/22xxx
0-1/2
C-1/2
The quartic term in Y in the optimization.
8 is constant under the decorrelation constraint and can be dropped from
We now introduce auxiliary matrix variables W and M, resulting in:
L(W,M, YERUXT WERRxd MERaxu
min
min max
1 where (W. Tr ,
MY) + Tr (WT, W A -2 M)
To verify the equivalence between the minimization problem (8) and the min-max problem take partial derivatives of L(W, M, Y) with respect to W (resp. M) and note that the minimum (resp. maximum) is achieved when W A (resp. I YY La). Substituting optimal W and M leads back to (8)
Finally, interchanging the order of minimization with respect to Y, with the optimization with respect to W and M, yields
min W WE M T
min L(W, M, Y).
(10)
The interchange is justified by saddle point property of L(W, M, Y) with respect to Y and M 44
5

---

4.2 Gradient optimization in the offline setting
In this subsection, we optimize the objective ( the offline setting, where the entire data matrix X is accessible. In this case, we solve the man-max problem (9) by alternating optimization steps. For fixed W and M, we minimize the objective function L(W, M, Y) over Y, which yields the relation
Y == arg min C(W,M,, = W YER
(11)
Before applying gradient optimization steps of the objective function L(W M, Y) with respect to W and M, we first simplify T appearing in 10, as part of the term, Tr [WT. W A 2)
r = =C
C
1/2 x 2 C/1/2 C1/2
1 T
Σ xxxx
(12)
T
where O2 = ||C² 1/2 x is the squared norm of the whitened data. As the transformation from whitened data to the recovered sources is orthogonal rotation, the squared norm of the sources and of the outputs is preserved:
a2
|
1/2x/
x1 ||sr||2 -1yee12
(13)
We then use
13 to rewrite
12
) ddiag Y Y)X +
Σ ly l |xxxx]
(14)
where ddiag(-) represents a diagonal matrix which keeps only the diagonal elements of the argument matrix.
We now obtain the update rules for W and M by gradient-descen ascent on (10) and by replacing
r-1 according to
W W
27
YXX
- A wT
W
2n (YX A ---x ddiagi (Y) T
(15)
1 YY
M M
Ls
(16)
Here is the ratio between the learning rates for W and M, and (0, is the learning rate for W, ensuring that M remains positive definite given a positive definite initialization.
4.3 Online algorithm
We now solve the min-max objective (9) in the online setting. At each time step, 4, minimize over the output, yr. by repeating the following gradient descent steps until convergence:
Yt - Yt + 7(c, - My.)
(17)
where y a small step size, and we have defined the projection C; = WX,, with biological interpretation described in Sec As in (TT). the dynamics converge to yo = M- We now take stochastic gradient descent-ascent steps W and M. We thus replace the averages Eqs. [15 16) with their online approximations
YX
yxx,
1
-
YY y.y
1 -2 ddiag Y YJX + - Cx
T
T
This yields our online ICA algorithm (Algorithm)
1 and the NN, see Section 5
5 Biological interpretation and neural implementation
We now show that our online ICA algorithm (Algorithm T maps onto an NN with local, activity- dependent synaptic update rules, which emulate aspects of synaptic plasticity observed experimentally.

---

Algorithm A similarity-preserv algorithm for Independent Component Analysis.
input data (X "I,,...,7T] ): dimension
output (V1..... yr dimension d
> estimated sources
initialize the matrix W, and positive definite matrix M.
fort = 1.2...., do
C, f Wx, run the following until convergence:
projection of inputs
dy.(7)
dy Ct My.(7)
neural dynamics
W t W + 2n(y: llxell c ; M + M+ (y.y! - -Ia) synaptic updates
end for
5.1 Neural architecture and dynamics
Our algorithm can be implemented by a biologically plausible NN presented in Fig. B.. The network consists of input layer of d neurons, representing the input data to be separated into independent components, and an output layer of neurons, with separate dendritic and somatic compartments, estimating the unknown sources. The network includes a of feedforward synapses between the inputs and the dendrites of the output neurons as well as a set of lateral synapses between the output somas Fig. B3.
Although wwo-compartment neurons have not been common in machine learning, in neuroscience, they often used model pyramidal cells the most numerous neuron type in the neocortex. Such neuron consists of an apical dendritic compartment, as well as somatic compartment, which have distinct membrane potentials [56 57
58, 59 Recently, such two-compartment neurons appeared in 63
bio-inspired machine learning algorithms 60 61
At each time step f, the network computes in two phases. First, the mixture x,, represented in the input neurons, is multiplied by the weight matrix W encoded by the feedforward synapses connecting the input neurons to the output neurons. This yields the projection C, Wx, computed in the dendritic compartments of the output neurons and then propagated to their somatic compartments.
Second, the d-dimensional output signal is computed somatic activity in the output neurons and corresponds to the estimated sources. This is accomplished by the fast recurrent neural dynamics in lateral connection, Eq. (17). converging to the equilibrium value assignment of in Algorithm [
The lateral synapses Fig. IB implement only the off-diagonal elements of M. Whereas diagonal elements of M would correspond to autapses (self-coupling of neurons), such Hebbian/ani- networks can be designed without them 44 45
5.2 Synaptic plasticity rules
To highlight the locality of our learning rules, we rewrite the element-wise synaptic updates for W and M in Algorithm Dranhhmmusiii Jusing sub-/super-scripts:
7 ci x ; Mij Mi - (vivv 615)
Wig + Wij + 21/ sí llvell"
- , 1siiisd (18)
In Eqs. (8). xo. is the activity of the j input neuron, si is the activity of the 8t output neuron, and of is the dendritic current of the 5th output neuron, all at time t. Furthermore, the influence of the dendritic current of synapse's strength is modulated by the term ||y:|| representing the overall activity of the output neurons.
How could total output neuronal activity be signaled to each feedforward synapse in the network? There are several diffusible molecules in the brain which may affect synaptic plasticity and whose concentration may depend on the overall neural activity. These include extracellular calcium [33]
GABA 29 65), dopamine (30 66 67 noradrenaline [1 688 D-Serin
70 or nitrie oxide 36 37 Finally, local field potential can also
(NO), although its range of action is contested [35|
affect synaptic plasticity 34. For the learning rule to function the online setting signaling must be
fast, requirement favoring local field potential and extracellular calcium out of the above candidates.
7

---

The learning rule (18) for feedforward synaptic weights, W, simplifies significantly near the optimum
of the objective, M Lg, and for the converged output activity, C, = Wx; My:
AW T/ (I- Iyrl A MM) yix; n (I ly PA y
(19)
Such an update a nonlinearly modulated Hebbian learning rule where the sign of plasticity changes with the total output activity. For low total output activity, ||y; < A₃. the update is Hebbian, i.c. long-term potentiation (LTP) for correlated inputs and outputs. For high output activity, I|y+ X²
the update is mebi-He i.e., long-term depression (LTD) for correlated inputs and outputs. We compare and contrast this global acpivity-dependnt modulation of plasticity with the Bienenstock, Cooper, and Munro (BCM) rule in subsection 5.3
Whereas fhree-factor learning has been invoked in multiple computational, especially reward-based, models (71 73 74| our model is the first to propose such learning in the fully normative approach
for ICA.
5.3 Comparison with existing rules
To understand the distinctive features our model versus existing approaches, we compare and contrast it with three existing models: 1. Oja's learning rule, 2. BCM learning rule [75), and 3. error-gated Hebbian rule (EGHR) [69].
19 generalized the original Oja's rule, Eq. 3.. with a component-wise nonlinear function g(:) as AW = o(y,)x; - (yy.) TW. However, this model and follow-up work inherited the main drawbacks of the standard Oja's rule, i.e., they require pre-whitening of the data and rely on non-local learning rules. Indeed, the last term of the learning rules of nonlinear Oja implies that updating the weight of synapse requires precise knowledge of output activities of all other neurons which are not available to the synapse (cf. 777 for details on standard PCA rules and networks of nonlinear neurons (78).
1. Nonlinear Oja's learning rule. 76
2. BCM learning rule. Switching of the sign of plasticity depending on the total output activity Eq. (19) is reminiscent of the BCM rule. It was initially postulated and later connected to an objective function 779 [0] characterizing the deviation from Gaussian distribution but mainly focusing on skewness rather than kurtosis as in our model. For correlated input and output, the BCM rule induces LTD for "sub-threshold" responses and LTP for three--trreshol" responses, with the threshold being a function of average output activity. Unfortunately, multiple output BCM neurons respond to the same dominant feature, producing an incomplete, highly redundant code 779 80 In contrast, our network has lateral inhibitory connections whose weights are updated via anti- rule Eq. (18) leading to the recovery of multiple sources. Although experimental evidence has validated BCM-like plasticity in parts of the visual cortex and the hippocampus, other brain areas have yet to show similar behavior. Interestingly, an "inverse" BCM rule, similar to ours has been proposed in the cerebellum
88 83.
3. Modulated Hebbian rules. Recent neural implementations of ICA 69 74 70
also introduced with Eu a constant, E(.) nonlinear function of the total activity, and g()) a component wise nonlinear function. This learning rule shares many similarities with ours. The term Es is constant characterizing the source distributions, which could identify with our X; terms, and the function E(-) resembles our lly: 1|2 but model dependent in their approach. This is where the similarities end as their objective function is inspired by the information itormation--hhoretiic framework [7 and ours by the insight from the FOBI procedure [27 84)
modulated Hebbian rules: AW- = (Eu - Ely:))y(yy))
and spectral methods from the SM method [44].
Their model can be considered partly normative since the neural architecture is predetermined and uses dend-de computinn neuron to determine the global modulating factor rather than having been derived from an optimization problem. Interestingly, their model does not use lateral connections for output decorrelation resulting in a model without direct interaction between outputs. The presence of pairwise inhibitory interaction crucial for our algorithm, leading to a globally optimal solution when the sources have distinct kurtosis. Numerical and theoretical analysis of the performance of the EGHR algorithm relies on the source distributions being the same and resulting in several equivalent optima.
8

---

6 Numerical simulations
In this section, we verify our theoretical results in numerical experiments. We use our model to perform ICA on both synthetic and real-worid datasets. We designed three sets of experiments to illustrate the performance of our algorithm. In the first set, Fig. 24, we used as sources artificially generated signals, in the second natural speech signals. Fig. 2B, and in the third natural scene images, Fig. 2C. According to the generative model, Eq. (T), we then used random full rank square mixing matrices, A, to generate the observed mixed signals, x,. From X,, we aimed to recover the original sources. We show that our algorithm recovers sources regardless of sub- or super-Gaussianiie of the kurtosis, which is essential for natural datasets. For details on the parameters used, see Appendix C.
Synthetic data. We first evaluate our algorithm on a synthetic dataset generated by independent and identically distributed samples. The data are generated from periodic signals, i.e., equare-periodic sine-wave, saw-tooth, and Laplace random noise. The data were chosen with the purpose of including both super- and sub-Gaussiae distribution known respectively as leptokurtic ("spiky", e.g.. the Laplace distribution) and platykurtic "fllatttoppd the three other source signal). We show in Fig. 2 the mixed signals in black, on the left plots. We show on the right plot the recovered sources, red, overlapped with the original sources, in blue, and the residual in green. We also show the histogram of each signal on the right side of each plot. Results are shown for 300 samples. We observe that the recovered and true sources nearly perfectly overlap, explaining the low value of the residual, which shows the almost perfect reconstruct performed by our algorithm.
In Appendix D. we provide a numerical comparison of the performance of our algorithm with competing models, namely, Herault-Jutten algorithm [85), EASI algorithm [86] 87,. Bell and Se- jnowski's ' algorithm [12.. the Amari algorithm [3], and finally nonlinear Oja algorithm [88] 76). These models were designed with NNs in mind and are seminal works on neural ICA algorithms. However, like nonlinear Oja algorithm, which is mentioned in Section 5.3.1, these models suffer from biological implausibility.
brief, our model either outperforms or competitive with the models mentioned above. These results also confirm that our algorithm can deal with combinations of sub- and Super-Gaussian sources, with or without pre- pre-whitening of the data.
Real-world data: Speech signals. For the audio separation task, we used speech recordings from the freely available TSP data set [89 recorded at 16kHz. The first source we use was obtained from a male speaker (MA02 04, wav), the second source from a female speaker (FA01 03. 03.wav) and the third source synthetically generated from uniform noise, as was previously used in the literature [90]. We show our results in Fig. 2B. We show the mixtures, the true sources, the recovered sources, and the residual. It is clear from the figure that our algorithm's 's outputs recover the true sources similarly to the synthetic dataset.
Real-world data: Natural scene images. We finally applied our algorithm to the task of recovering images from their mixtures, on data already used for BSS tasks 21,91,27 as shown in Fig. 2 Here, we show separately the original sources, top images, the mixtures, middle images, and the recovered sources, bottom images of Fig. DC We considered three grayscale images of size 256 x 512 pixels (shifted and scaled to have zero-mean and unit variance), such that each image is treated as one source, with the pixel intensities representing the samples. We again observe in Fig, 2 that the recovered sources are nearly identical to the original sources. We can also see that the histograms of the recovered sources nearly match the histograms of the original sources, up to their sign.
7 Discussion
We proposed a new single-layer ICA NN with biologically plausible local learning rules. The normative nature of our approach makes the biologically realistic features of our NNs readily interpretable. In particular, our NN uses neurons with two separate compartments and is trained with extended Hebbian learning rules. The changes in synaptic strength are modulated by the total output neuronal activity, equivalent to performing gradient optimization of our objective function. We demonstrated that the proposed rule reliably converges to the correct solution over wide range of mixing matrices, synthetic, and natural datasets. The broad applicability and easy implementation
I Freely available at htt:://ww.mmss cce.esppppppceec..eeeee..c www.e..willlcccooo wwregill.ca/Dccunnn cww.aeettteeee/eaaaaa/aa// (Accessed May 24th 2021). FFeely available at https//rrsearchhics aalto www.aaalwo..iicaaaaaaaaaaaaaaaae (Accessed May 24th 2021).

---

Minturn:
Recimered & true
B
0.0
-0.5
-0.0
Renovared &
Recovered
Recovered
Mistum
Recowerdd true
Squa
N MM
0.5
10000
20500
30000
1960
20010
30000
J0000
20000
30000
Sinture
Recovered true
Source
Source
Source
Sgral
-.5
Misture
Recovered true
Mixture
Mixture 2
Mixture 3
-0.
Recovered
Recovered
Recovered
Sumplicr
Sumples
by Alg.1
Teae sourcss
Reddual
Recnvered Tras
Figure Our ICA algorithm recovers independent sources from synthetic and real-world mixtures. A. Synthetic data, B. Natural speech data, C. Natural image data. In A-B. mixed signals are shown in black, the recovered signals in red, the true sources in blue, and their residual difference in blue. We also show the associated distributions. C. The sources, mixture, and recovered sources, in top, middle and bottom rows respectively.
of our NN and learning rules could further advance neuromorphic computation reveal the principle underlying BSS computation the brain.
92 93 94 and may
Recent work on canonical correlation analysis [95] 96,
98 and ICA- of which rely two-compartment neurons [103] 51]. These NNs could, in principle, be used for popular BSS tasks
97], slow feature analysis
like algorithms have led to biologically plausible NNs (99
100 [OI [02],
107 in the context of kernel ICA [09]. This suggests the existence of a single model of ww--oomprttment neurons and non-trin non-trivial local learning rules for BS. In future work, we aim proposing such model, including high-order statistics, temporal correlation, and diversity of views.
known second-order blind identification [T04] 105
106
One limitation of our approach is the inability of the model to separate sources with the same kurtosis. Yet, as long as sources possess some distinct even-order moments, our scaling rule can be altered to separate the sources (27]. Another limitation is the well-known sensitivity of kurtosis outliers. This limitation could be overcome if scaling varies as sublinear function of the total activity [40] the locality of the learning rules.
These changes do not affect the neural architecture
Clarifying the limitations of our model leads us to ask various follow-up questions left for future work. How we further generalize the solution beyond the choice of nonlincarity and beyond the task of linear ICA? We could envision considering more than two covariance matrices as in the JADE algorithm [T1O, [I) which effectively performs joiol-diagonalizzatonn of arbitrarily many matrices. A neural solution was proposed in [113] but again relies on non-local Oja- based rules. Ongoing work nonlinear ICA [114 [15] is of great interest to since it might be perfect candidate for multi-layered architectures.
Recently, several works proposed biologically plausible supervised learning algorithms 6
63
53 [16]. Combining these with ICA and unsupervised learning algorithms general would provide more comprehensive description of cognitive processes.
10

---

Acknowledgments and Disclosure of Funding
Y.B. is grateful Romain Cosentino, and Claudia Skok Gibbs for insightful discussions related to this work and feedback on this manuscript. We also thank the members of the Neural Circuits and Algorithms Group the Flatiron Institute for providing feedback on an early version of this work.
The authors did not receive any third party funding for the completion of this project.
References
[1] Robert Desimone and John Duncan. Neural mechanisms of selective visual attention, Annual Review of Neuroscience, 18(1):193-222, 1995.
[2] Rachel Wilson and Zachary F Mainen. Early events in olfactory processing. Annual Review of Neuroscience, 29:163-201 2006.
[3] Mark A Bee and Christophe Micheyl. The cocktail party problem: what is it? how can it be solved? and why should animal behaviorists study it? Journal of comparative psychology, 122(3):235, 2008.
[4] Josh H McDermott. The cocktail party problem. Current Biology, 19022)RR124-RRO0 2009.
[5] Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent component analysis and applications. Academic press, 2010.
[6] Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-
314, 1994.
[7] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks, (5(-5))411-4430 2000.
[8] Anthony Bell and Terrence Sejnowski. Edges are the 'independent components' of natural scenes. Advances in Neural Information Processing Systems, 9:831-837, 1996.
[9] Anthony Bell and Terrence Sejnowski. The "independent components" of natural scenes are edge filters. Vision research, 37(23):3327-333 1997.
[10] Michael Lewicki. Efficient coding of natural sounds. Nature neuroscience, 5(4):356-363, 2002.
[11] Jitendra Sharma, Alessandra Angelucci, and Mriganka Sur. Induction of visual orientation modules in auditory cortex. Nature, 404(6780))841-84777 2000.
[12] Anthony J Bell and Terrence J Sejnowski. An information-maxinnne separation and blind deconvolution. Neural Computation, 7(6):1 19t-1129--1199 1159, 1995.
approach to blind
[13] Shun-Ichi Amari, Andrzej Cichocki, and Howard Yang. A new learning algorithm for blind signal separation. Advances in Neural Information Processing Systems, 8:757-763, 1995.
[14] Dinh Tuan Pham and Philippe Garat. Blind separation of mixture of independent sources through maximum likelihood approach. In In proc. eusipco. Citeseer, 1997.
[15] Te- Lee. Independent component analysis. In Independent component analysis, pages 27-66. Springer, 1998.
[16] Ton Lee, Mark Girolami, Anthony Bell, and Terrence J Sejnowski. A unifying information-theoreiic framework for independent component analysis. Computers Mathe- matics with Applications, 39(1 1):1-21, 2000.
[17] Juha Karhunen, Erkki Oja, Liuyue Wang, Ricardo Vigario, and Jyrki Joutsensalo. A class of neural networks for independent component analysis. IEEE Transactions on Neural Networks, 8(3):486-504, 1997.

---

[18] Shun-Ichi Amari and Andrzej Cichocki. Adaptive blind signal processing- neural network approaches. Proceedings of the IEEE, 86(10):20662044 1998.
[19] Aapo Hyvärinen and Erkki Oja. Independent component analysis by general nonlinear Hebbian-like learning rules. Signal Processing, 64(3)300-331,, 1998.
[20] Andrzej Cichocki, Juha Karhunen, Wlodzimierz Kasprzak, and Ricardo Vigario. Neural networks for blind separation with unknown number of sources. Neurocomputing. 24(1-3):55-
93, 1999.
[21] Peter Földiak. Forming sparse representations by local anti-Hebbian learning. Biological Cybermetics, 64(2):165-170, 1990.
[22] Ralph Linsker. A local learning rule that enables information maximization for arbitrary input distributions. Neural Computation, 1661-666,, 1997.
[23] Cengiz Pehlevan, Sreyas Mohan, and Dmitri B Chklovskii. Blind nonnegative source separation using biological neural networks. Neural Computation,
2017.
[24] David Lipshutz and Dmitri B Chklovskii. Bio-NICA: A biologically inspired single-layer network for nonnegative independent component analysis. arXiv preprint 2xXi::0010.1263: 2020.
[25] Berfin Simsek and Alper T Erdogan, Online bounded component analysis: simple recurrent neural network with local update rule for unsupervised separation of dependent and indepen- dent sources. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 1639-1643. IEEE, 2019.
[26] Alper T Erdogan and Cengiz Pehlevan. Blind bounded source separation using neural networks with local learning rules. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3812-3816. IEEE, 2020.
[27] J-F Cardoso. Source separation using higher order moments. In ICASSP, pages 1109-2112. IEEE, 1989.
[28] Jari Micttinen, Sara Taskinen, Klaus Nordhausen, and Hannu Oja. Fourth moments and independent component analysis. Statistical Science, 30(3):372-390, 2015.
[29] Tatsuya Hayama, Jun Noguchi, Satoshi Watanabe, Noriko Takahashi, Akiko Hayashi-Takan Graham CR Ellis-Davies, Masanori Matsuzaki, and Haruo Kasai. GABA promotes the com- petitive selection of dendritic spines by controlling local ca 2+ signaling. Nature neuroscience, 16(10):1409, 2013.
[30] John NJ Reynolds, Brian Hyland, and Jeffery R Wickens. cellular mechanism of reward- related learning. Nature, 413(6851):667700 2001.
[31] Humberto Salgado, Georg Köhr, and Mario Trevino. Noradrenergic 'tone' determines di- chotomous control of cortical spike-timing-deee timing-dependent plasticity. Scientific reports, 2(1):1-7, 2012.
[32] Christian Henneberger, Thomas Papouin, Stéphane HR Oliet, and Dmitri A Rusakov. Long- term potentiation depends on release of d-serine from astrocytes. Nature, 233 2010.
[33] OJMD Coenen, DM Eagleman, V Mitsner, TM Bartol, AJ Bell, and TJ Sejnowski. Cerebellar glomeruli: Does limited extracellular calcium direct a new kind of plasticity. In Society for Neuroscience Abstracts, volume 27, 2001.
[34] Matti Weckström and Simon Laughlin. Extracellular potentials modify the transfer of informa- tion at photoreceptor output synapses in the blowfly compound eye. Journal of Neuroscience, 30(28):9557-9555 2010.
[35] Emily Huang. Synaptic plasticity: a role for nitric oxide LTP. Current Biology, 7():R141- R143, 1997.
12

---

[36] Christian Hölscher. Nitric oxide, the enigmatic neuronal messenger: its role in synaptic plasticity. Trends in neurosciences, 20(7):298- 303, 1997.
[37] Neil Hardingham, James Dachtler, and Kevin Fox. The role of nitric oxide in pre-synaptic plasticity and homeostasis. Frontiers in cellular neuroscience, 7:190, 2013.
[38] Karl Pearson. On lines and planes of closest fit systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):59--72, 1901.
[39] Ian T Jolliffe. Principal Component Analysis and Factor Analysis, volume 1. Springer, 1986.
[40] Kun Zhang and Lai-
Lai-Wan Wan Chan. ICA by PCA approach: relating higher-order statistics to second-order moments. In International Conference on Independent Component Analysis and Signal Separation, pages 31 1-318. Springer, 2006.
[41] Erkki Oja. Principal components, minor components, and linear neural networks. Neural Networks, 566):927-935, 1992.
[42] A-J Van Der Veen, ED F Deprettere, and A Lee Swindlehurst. Subspace-based signal analysis using singular value decomposition. Proceedings of the IEEE, 1179):2777-330, 1993.
[43] Cengiz Pehlevan and Dmitri Chklovskii. A normative theory of adaptive dimensionality reduction in neural networks. In Advances in Neural Information Processing Systems, pages 2269-2277,2011 2015.
[44] Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri B Chklovskii. Why do similarity matching objectives lead to Soobiannanti-Hebbiie networks? Neural Computation, 30(1):84 -12444444 2017.
[45] Victor Minden, Cengiz Pehlevan, and Dmitri B Chklovskii. Biologically plausible online principal component analysis without recurrent neural dynamics. In 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pages 104-111. IEEE, 2018.
[46] Cengiz Pehlevan and Dmitri B Chklovskii A Saabiann/ant-Heebbiae network derived from online non-negative matrix factorization can cluster and discover sparse features. In 2014 48th Asilomar Conference on Signals, Systems and Computers, pages 769-775. IEEE, 2014.
[47] Yanis Bahroun, Eugénie Hunsicker, and Andrea Soltoggio. Neural networks for efficient nonlinear online clustering. In International Conference on Neural Information Processing. pages 316-324. Springer, 2017.
[48] Anirvan Sengupta, Cengiz Pehlevan, Mariano Tepper, Alexander Genkin, and Dmitri Chklovskii. Manifold-iiling localized receptive fields are optimal in similarity-preserving neural networks. In Advances Neural Information Processing Systems, volume 31, 2018.
[49] Yanis Bahroun, Dmitri Chklovskii, and Anirvan Sengupta. trained on transformed images recapitulates salient features of the fly motion detection circuit. Advances in Neural Information Processing Systems, volume 32, 2019.
cenuilarity-prseervii network
[50] Cengiz Pehlevan, Xinyuan Zhao, Anirvan M Sengupta, and Dmitri Chklovskii. Neurons as canonical correlation analyzers. Frontiers in Computational Neuroscience, 14:55, 2020.
[51] David Lipshutz, Yanis Bahroun, Siavash Golkar, Anirvan M Sengupta, and Dmitri B Chklovskii. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation, 3339) 2352, 2021.
[52] Yanis Bahroun and Andrea Soltoggio. Online representation learning with single and multi- layer Hebbian networks for image classification. In International Conference Artificial Neural Networks, pages 354-663. Springer, 2017.
[53] Shanshan Qin, Nayantara Mudur, and Cengiz Pehlevan. Contrastive similarity matching for supervised learning. Neural Computation, 130--1333-1322 2021.
[54] Yanis Bahroun and Dmitri Chklovskii. A neural network with local learning rules for minor subspace analysis. arXiv preprint arXiv:2102. 2021.
13

---

[55] Kantilal V Mardia, John T Kent, and John M Bibby. Multivariate Analysis (probability and mathematical statistics). Academic Press London, 1980.
[56] Paul F Pinsky and John Rinzel. Intrinsic and network rhythmogenesis reduced Traub model for CA3 neurons. Journal of computational neuroscience, 1(1):39 60, 1994.
[57]
Sonia Gasparini and Jeffrey C Magee, State-dependent dendritic computation in hippocampal CA1 pyramidal neurons. Journal of Neuroscience, 26(7):2088 -10080 2006.
[58]
Nelson Spruston. Pyramidal neurons: dendritic structure and synaptic integration. Nature Reviews Neuroscience, 9(3):206- 221, 2008.
[59] Yael Katz, Vilas Menon, Daniel A Nicholson, Yuri Geinisman, William L Kath, and Nelson Spruston. Synapse distribution suggests two-stage model of dendritic integration in CA1 pyramidal neurons. Neuron, 63(2):171-177, 2009.
[60] Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. Elife, 6xe22901, 2017.
[61] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. Advances in Neural Information Processing Systems, 31:8721-8732, 2018.
[62] Blake A Richards and Timothy P Lillicrap. Dendritic solutions the credit assignment problem. Current opinion in neurobiology, 54:28-36, 2019.
[63] Siavash Golkar, David Lipshutz, Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. A simple normative network approximates local hebbian learning the cortex. Advances in Neural Information Processing Systems, 33, 2020.
[64] Spyridon Chavlis and Panayiota Poirazi. Drawing inspiration from biological dendrites to empower artificial neural networks. Current Opinion Neurobiology. 70:1-10, 2021.
[65] Vincent Paille, Elodie Fino, Kai Du, Teresa Morera-Herreras, Sylvie Perez, Jeanette Hellgren Kotaleski, and Laurent Venance. GABAergic circuits control spike-timing-dependenni plasticity. Journal of Neuroscience, 33(22))9333 93633, 2013.
[66] Ji- Ji-Chuan Zhang, Pak Pak-Ming Lau, and Guo- Qiang Bi. Gain in sensitivity and loss in temporal contrast of stdp by dopaminergic modulation at hippocampal synapses. Proceedings of the National Academy of Sciences, (3331))13028-13000
2009.
[67] Sho Yagishita, Akiko Poyashi-Takage Graham CR Ellis- EllieeDavies, Hidetoshi Urakubo, Shin Ishii, and Haruo kasai. A critical time window for dopamine actions the structural plasticity dendritic spines. Science, 345(6204):111-11200 2014.
[68] Joshua P Johansen, Lorenzo Ditz-Ma Hiroki Hamanaka, Takaaki Ozawa, Edgar Yeu, Jenny Koivumaa, Ashwani Kumar, Mian Hou, Karl Deisseroth, Edward S Boyden, et al. Hebbian and neuromodulatory mechanisms interact to trigger associative memory formation. Proceedings of the National Academy of Sciences, 11161))E5544-55552, 2014.
[69] Takuya Isomura and Taro Toyoizumi. A local learning rule for independent component analysis. Scientific Reports, 6(1):1-17, 2016.
[70] Takuya Isomura and Taro Toyoizumi. Error-gated Hebbian rule: A local learning rule for principal and independent component analysis. Scientific Reports, 8(1):1-11, 2018.
[71] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.
[72] Pietro Mazzoni, Richard A Andersen, and Michael I Jordan. A more biologically plausi- ble learning rule for neural networks. Proceedings of the National Academy of Sciences, 88(10):443344437, 1991.
[73] Ronald J Williams. Simple statistical gradient-followie algorithms for connectionist rein- foreement learning. Machine learning, 8(3-4):229-255 1992.
14

---

[74] Lukasz Kušmierz, Takuya Isomura, and Taro Toyoizumi. Learning with three factors: modu- lating Hebbian plasticity with errors. Current Opinion in Neurobiology, 46:170-177,
[75] Elie Bienenstock, Leon N Cooper, and Paul W Munro. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience, 2(1):32-48, 1982.
[76] Erkki Oja. The nonlinear PCA learning rule in independent component analysis. Neurocom- puting, 17(1):25-45, 1997.
[77] Cengiz Pchlevan and Dmitri B Chklovskii. peuuoociennc-inspiice online unsupervised learning algorithms: Artificial neural networks. IEEE Signal Processing Magazine, 36(6):88-96 2019.
[78] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381(6583)6607-609) 1996.
[79] Nathan Intrator and Leon N Cooper. Objective function formulation of the bem theory of visual cortical plasticity: Statistical connections, stability conditions. Neural Networks, 5(1):3-17, 1992.
[80] GC Castellani, N Intrator, H Shouval, and LN Cooper. Solutions of the bem learning rule in network of lateral interacting nonlinear neurons. Network: Computation in Neural Systems, 10(2):111-121, 1999.
[81] Michiel Coesmans, John Weber, Chris De Zeeuw, and Christian Hansel. Bidirectional parallel fiber plasticity in the cerebellum under climbing fiber control. Neuron, 17-00, 2004.
[82] Henrik Jörntell and Christian Hansel. Synaptic memories upside down: bidirectional plasticity at cerebellar parallel fiber-purkinje cell synapses. Neuron, 52(2):227-238, 2006.
[83] Kaspar E Vogt and Marco Canepari. On the induction of postsynaptic granule cell-Purkinje neuron LTP and LTD. The Cerebellum, 9()):884-29) 2010.
[84] Lucas Parra and Paul Sajda. Blind source separation via generalized eigenvalue decomposition. Journal of Machine Learning Research, 1:11-1126 2003.
[85] Christian Jutten and Jeanny Herault. Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture. Signal processing, 24(1):1-10, 1991.
[86] Beate Laheld and mance. In Proc. EUSIPCO, volume 1, pages 183-186. Citeseer, 1994.
Cardoso. Adaptive source separation with uniform perfor-
[87] J-F Cardoso and Beate H Laheld. Equivariant adaptive source separation. IEEE Transactions on signal processing, 44 14(12):3017-3033 1996.
[88] Juha Karhunen, Liuyue Wang, and Ricardo Vigario. Nonlinear pea type approaches for source separation and independent component analysis. In Proceedings of ICNN'95- 95-tnnteraaiionll Conference on Neural Networks, volume 2, pages 995-1000. IEEE, 1995.
[89] Peter Kabal. TSP speech database. McGill University, Database Version, 1(0):09-02, 2002.
[90] Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non-linear ICA. arXiv preprint arXiv:1710.05050, 2017.
[91] Aapo Hyvärinen and Patrik Hoyer. Emergence of phase-and shift-invariant features by de- composition of natural images into independent feature subspaces. Neural computation, 12(7): 2000.
[92] Jussi H Poikonen and Mika Laiho. Online linear subspace learning in an analog array computing architecture. In CNNA 2016; 15th International Workshop on Cellular Nanoscale Networks and their Applications, pages 1-2. VDE, 2016.
[93] Mohammed E Fouda, Emre Neftci, Ahmed Eltawil, and Fadi Kurdahi. Independent component analysis using RRAMs. IEEE Transactions Nanotechnology, 18:611-615, 2018.
15

---

[94] Cengiz Pehlevan. A spiking neural network with local learning rules derived from nonnegative similarity matching. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7958-7962. IEEE, 2019.
[95] J. D. Carroll. Generalization of canonical correlation analysis three or more sets of variables. Proc. APA, pages 227-228, 1968.
[96] David R Hardoon, Sandor Szedmak, and John Shawe-Taylor Canonical correlation analysis:
An overview with application to learning methods. Neural Computation, 16(12):2639-266 2004.
[97] Lucas C. Parra. Multiset canonical correlation analysis simply explained. arXiv preprint arXiv:1802.0377 2018.
[98] Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4):715-770 2002.
[99] Joel Zylberberg, Jason Timothy Murphy, and Michael Robert DeWeese. A sparse coding model with synaptically local plasticity and spiking neurons can account for the diverse shapes of VI simple cell receptive fields. PLoS computational biology, 7(10): 2011.
[100] Paul King, Joel Zylberberg, and Michael R DeWeese. Inhibitory interneurons decorrelate ex- citatory cells to drive sparse code formation in spiking model of V1. Journal of Neuroscience, $3(13):5475-543 2013.
[101] Sophie Denève, Alireza Alemi, and Ralph Bourdoukan. The brain as an efficient and robust adaptive learner. Neuron, 94(5):969 2017.
[102] Wieland Brendel, Ralph Bourdoukan, Pietro Vertechi, Christian K Machens, and Sophie Denéve. Learning to represent signals spike by spike. PLoS computational biology, 16(3)xe007692, 2020.
[103] David Lipshutz, Charles Windolf, Siavash Golkar, and Dmitri Chklovskii. A biologically plausible neural network for slow feature analysis. In Advances in Neural Information Processing Systems, volume 33, pages 14986-149999 2020.
[104] James V Stone. Blind source separation using temporal predictability. Neural Computation, 1357):159- 2001.
[105] Hailin Liu and Yiuming Cheung. A learning framework for blind source separation using generalized eigenvalues. In International Symposium on Neural Networks, pages 472-477. Springer, 2005.
[106] Tobias Blaschke, Pietro Berkes, and Laurenz Wiskott. What the relation between slow feature analysis and independent component analysis? Neural Computation, 18(10)2495-2208 2006.
[107] Claudia Clopath, André Longtin, and Wulfram Gerstner. An online Hebbian learning rule that performs independent component analysis. In Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20, Curran Associates,
Inc., 2008
[108] Francis Bach and Michael I Jordan. Kernel independent component analysis. Journal of Machine Learning Research, (aluul):1448,) 2002.
[109] Yi-Ou Li, Tülay Adali, Wei Wang, and Vince D Calhoun. Joint blind source separation by multiset canonical correlation analysis. IEEE Transactions on Signal Processing, 57(10)33918- 3929, 2009.
[110] Jean-Franccois Cardoso and Antoine Souloumiac. Blind beamforning for non-Gaussian signals. IEE Proceedings F (Radar and Signal Processing), 140())362-370 1993.
[111] Hannu Oja, Seija Sirkiä, and Jan Eriksson. Scatter matrices and independent component analysis. Austrian Journal of Statistics, 35 2006.
16

---

[112] Esa Ollila, Hannu Oja, and Visa Koivunen. Complex-valued ICA based on pair of generalized covariance matrices. Computational Statistics & Data Analysis, 52(7):3789-3805, 2008.
[113] Ch Ziegaus and Elmar Wolfgang Lang. A neural implementation of the JADE algorithm (nJADE) using higher-order neurons. Neurocomputing, 56:79-100, 2004.
[114] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Tumne. Nonlinear ICA using auxiliary variables and generalized contrastive learning. The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[115] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[116] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, pages 1-10, 2021.
17


Reviews:
[{"review_id": "l93o7exOZmB", "reviewer": "Reviewer_ocjn", "summary": "The authors propose a compartmental single-layer neural network model to be used in an online ICA algorithm utilizing a local synaptic update rule. The novel algorithm is based on the FOBI procedure for PCA by restating the objective function using similarity matching. As opposed to hebbian update rules, the synaptic updates used in the present paper incorporate a time-varying modulating factor which is computed as a function of the total output activity. Biological plausibility of this relaxed locallity constraint is justified by hinting at experimential results showing the importance of a third factor in the outcome of hebbian plasticity. The approach is verified using three different experiments including synthetic and real-world data.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe search for a local learning rule implementing ICA has a long history and many approaches have been investigated in the past. The presented approach is comparable with the work of Isomura and Toyoizumi which is also noted in the paper. While the mathematical details elude the reviewer, the implications and resulting learning rules seem quite similar. The main difference appears to be trading off a constant \"error baseline\" for lateral inhibitory connections. A more thourough comparison of the two approaches could further improve the quality of the paper.\n\nQuality: \nThe submission seems to be technically sound. Claims are supported by mathematical proofs and adequate experimental results. The authors are honestly stating the weaknesses of their approach.\n\nClarity: \nThe approach is laid out in a clear and understandable fashion. Included figures are helpful for understanding the resulting NN structure and the correspondence to the FOBI procedure.\n\nSignificance:\nIntroducing yet another biologically plausible method for ICA might be considered of mediocre importance. Nonetheless, the work may advance our understanding of the inner workings of biological neural networks by linking a novel objective function to a simple network motif.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and possible remedies are listed. Due to the specific mathematical nature of the the paper its societal impact is unlikely to exceed the impact of the field as a whole.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "TPrT6hCNk9w", "reviewer": "Reviewer_EXE1", "summary": "The paper proposes a biologically plausible neural network (BPNN) with an independent component analysis (ICA) objective loss function to solve blind source separation problem. The main contribution of this paper lies in that, it extends BPNN for principal component analysis (PCA) [56] to BPNN for ICA with the fourth-order blind identification (FOBI). Simulation experiments demonstrate that the proposed approach is effective in solving blind source separation problem. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The propose approach is interesting and has innovations both theoretically and algorithmically. The paper is well organized and presented. \n\nOne shortage of the paper lies in that the experiment part is not strong. The experiments contain three datasets of a synthetic data, a speech data, and an image data. The results are mostly presented with illustrations rather than quantitative criterions. It also lacks of comparisons with other approaches. Adding quantitative evaluations and comparisons can help demonstrate the advantages of the proposed approach.\n\nIt is also interesting to ask what are the possible applications of the proposed approaches? What can we benefit from the biologically plausible neural network in the blind source separation problem? Can the proposed model help explain biological mechanisms in the brain?\n\n============================\n\nPost-rebuttal:\n\nThe authors have addressed most of my comments. I have raised my score from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QsOJZGEYukY", "reviewer": "Reviewer_yveC", "summary": "This is a well-written paper containing novel ideas. The paper is motivated by an interesting question: how might fundamental computations such as independent component analysis (ICA) be performed by brains? The authors develop a neat algorithm to perform ICA. The paper claims that the developed algorithm is biologically plausible, i.e. might run in a similar form in a biological brain. \n\n# Update (2021-08-20)\n\nFollowing the author's response from 2021-08-20, all my questions are answered. I had misunderstood the semantics of $x_t$. I updated my score. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Update (2021-08-20)\n\nFollowing the author's response from 2021-08-20, all my questions are answered. I had misunderstood the semantics of $x_t$. I updated my score. \n\n\n# Overview\n\nThis is a well-written paper containing novel ideas. The paper is motivated by an interesting question: how might fundamental computations such as independent component analysis (ICA) be performed by brains? The authors develop a neat algorithm to perform ICA. The paper claims that the developed algorithm is biologically plausible, i.e. might run in a similar form in a biological brain. \n\nMy main criticism of the paper is that unless there is a misunderstanding on my part, the described algorithm may not be biologically plausible. I would be grateful if the authors could respond to my questions Q1, Q2 and Q3 below. If the authors argue successfully in this review process and in the paper that the proposed algorithm is biologically plausible, I will change the review score that I am assigning today. \n\n# Questions for the authors\n\n- Q1. Unless I misunderstood the paper, the algorithm that the authors propose and claim to be biologically plausible does not appear biologically plausible to me. In Algorithm 1 and Section 5.1, the authors explain that computing independent components involves updating the “weight matrix W encoded by the feedforward synapses connecting the input neurons to the output neurons”. This would imply that synapses re-learn completely new weights every time that the set of neurons computes independence components. It would be very surprising if the brain worked in such a manner. To be concrete: while listening to a conference speaker in a full conference room, my auditory system constantly has to separate the speaker’s voice from the background noise. As this computation happens, my auditory system’s synapses do not need to re-wire themselves in every fraction of a second, as I process a new chunk of audio signal. Instead, my auditory system in the brain developed some kind of noise filter when I was an infant, effectively learning synaptic weights that let me separate human voice from background noise across a wide range of environments. These weights may have changed somewhat over the years, but they certainly do not need to update every moment that I am processing auditory information. In essence, the authors are using (synaptic) weights that we know to learn slowly as fast-updating working memory. A biological neuron cannot learn fast enough to support running Algorithm 1 in real-time. \n\n- Q2. It would be useful if the authors could add details on how the proposed algorithm would map to known structures in say the human brain. The authors mention that “Various brain areas, such as the early visual, auditory, and olfactory systems, are known to identify 17 sources from their mixtures effortlessly [20, 85, 5, 52], an unsupervised task also known as blind 18 source separation (BSS) [19].” Since NeurIPS is mainly a machine learning conference, it would be help if the authors could expand on this sentence, survey in more detail what relevant information is known, and describe where they see the proposed algorithm fit in. For say visual processing, would the proposed algorithm run in the neurons of the primary visual cortex (V1)? \n\n- Q3. Have the authors benchmarked against any other ICA implementations? \n\n\n# Other suggestions and ideas for improvement\n\n\n- Algorithm 1 contains notation that I find odd: just after “run the following until convergence:”, it looks like we assign to a derivative. Is that a typo? The line should probably be something closer to equation (16). \n\n\n- The proposed algorithm is very specific. Is there biological evidence suggesting that the brain is running an algorithm like the one proposed? \n\n- Reference [5] cited by the authors suggests that auditory scene analysis might involved not only the bottom-up kind of computing that is present in the proposed algorithm, but also top-down processing: ”In humans, auditory scene analysis involves both bottom-up and top-down processes (reviewed in Bregman, 1990; Carlyon, 2004; Feng & Ratnam, 2000; Näätänen, Tervaniemi, Sussman, Paavilainen, & Winkler, 2001). Bottom-up mechanisms are “stimulus-driven,” meaning that they operate only or primarily on cues present in the acoustic signal itself; they are largely automatic and obligatory, meaning that they do not critically depend on attention (although this is currently a disputed issue). In contrast, top-down processes depend on a listener's prior experience and expectations, and thus involve higher-level cognitive processes, such as learning, memory, and attention.” It would be useful to point out that a bottom-up algorithm like the one described in this paper would likely only constitute one part of a more complex system to perform e.g. signal processing computations. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "O_dWB4-OALT", "reviewer": "Reviewer_h9nv", "summary": "The authors propose a biologically plausible neural network implementation for online ICA. It is based on a similarily objective with inspiration from the FOBI procedure. The authors derive an online learning rule and interpret the terms in the updates in a synaptically local manner with the addition of a global activity level signaling mechanism. The authors then validate the method in synthetic, speech, and image datasets.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is clearly written, the derivations are sound as far as I can tell, and the results show the applicability to synthetic and real datasets. The results are of interest to readers at the intersection of machine learning and computational neuroscience and possibly hardware implementations of neural networks.\n\nAlthough I have not run it, the provided code looks like it will reproduce the results from the paper.\n\nMost of my comments are related to clarifying certain points in the paper.\n\n- For the references on lines 111-114, it would be helpful to state whether or not the application of Similarity Matching lead to local or online learning rules for any of those models. Are any of the particular choices that lead to a local, online algorithm for BSS generally applicable? Those references share the most conceptual overlap with this manuscript and it would help to have some clarificaltion on how much technical overlap there is across methods/models.\n- Since $||y_t||$ is not clearly a local variable, it requires some interpretation in terms of a potential global modulatory component, which the authors provide. One point that could use some discussion is about how to match timescales since $||y_t||$, at least in the current presentation, needs to modulate at the data sampling rate. Does this point towards an interpretation of faster modulating processes from lines 214-219 being a better potential biological counterpart?\n- The difficulty of using Oja’s rule for FOBI on lines 95-97 could be more clear, particularly the sentence that starts on line 95 (what does \"true output\" mean?). Is another way of stating the problem that Oja’s rule could be applied to $z$ after it is computed, but the goal is to learn a $W$ that applies to $h$?\n- Are the results presented in the main text for the offline or online version of the algorithm?\n\n- Related to limitation 1 below, I would have expected the online algorithms to have some amount of “burn-in” time where the errors were higher before they become adapted to the statistics of the dataset, but this is not apparent in the figures. Is this masked by the trial shuffling for training and then unshuffling for display? Can you show (in the appendix) how fast the online variant’s error converges toward the offline solution?\n- How fast does the iteration in Eq 16 converge?\n\n- In the second paragraph of the introduction, it might make sense to have a sentence about biologically plausible networks that learn other models similar to ICA, for instance refs [46], [48], and \n    - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007692\n    - https://www.sciencedirect.com/science/article/pii/S0896627317304178\n    - https://www.jneurosci.org/content/33/13/5475.short\n    - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002250\n\nResults on data\n- For Fig 2 A, I think it would be more clear to group the 4 mixtures on the left and the 4 recovered sources on the right. Alternating them gives the impression that there is some connection between mixture 1 and source 1, mixture 2 and source 2, etc.\n\nMinor comments\n\n- 280: reconstruct -> reconstruction\n- Fig 2 caption: second sentence needs correction", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. One algorithmic mismatch is the desire to have “online” be a requirement for biologically plausibility, but then time-shuffling the training examples to remove temporal correlations (if I’m understanding Appendix B correctly). Ideally, an online biologically inspired learning algorithm should be applicable to streaming data which may have temporal correlations. Achieving both an online + synaptically local network and learning from correlated timeseries may be beyond the scope of the paper, but it should be addressed at least in the Discussion as a limitation/future direction.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Lo4kFGhGGAt", "reviewer": "Reviewer_QCCE", "summary": "This work proposes a novel similarity-preserving objective function for targeting ICA. They utilize this objective function to derive a gradient descent-based learning algorithm that takes advantage of this objective function. This algorithm is mapped to a single-layer neural network that is able to separate independent sources without pre-processing. They compare the proposed learning algorithms with oja-based methods, BCM, and Modulated Hebbian learning.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this work are summarized as follows:\n- The work proposes a novel perspective on a topic that has not been heavily explored.\n- The development of biologically plausible neural networks could provide valuable insight into the brain’s computational properties. In addition, a biologically plausible method of ICA is particularly valuable.\n- The theoretically grounded algorithm successfully integrates insights from the FOBI procedures into a neural network structure. The experimental results indicate that the implemented neural network can successfully perform ICA.\n\nHowever, the observed weaknesses of the paper are summarized as:\n- The experimental results presented demonstrates that the methodology can recover the original sources from a Mixed signal. But it is difficult to quantify the degree of that success for comparison with other methodologies. The authors should include a quantitative evaluation of the algorithm's effectiveness and quality of the recovered signals.\n- While the work contrasts the proposed work with the Oja-based, BCM and Modulated Hebbian methodologies, the work does not present experimental comparisons to highlight the benefits of the proposed approach. The authors should highlight these distinctions.\n- The authors utilize the proposed scheme only in a single layer architecture. The authors should demonstrate the scalability of the proposed scheme using a deeper model. \n\nIn addition to the above comments, we hope the following can help the authors further improve their work.\n- The font size of some text in both figures should be increased.\n- While computational restrictions are not necessarily critical in the target scenario, it would be good to provide such information to evaluate the algorithm’s overhead properly.\n- Since the two-compartment neural model is uncommon, it would be good to include a concrete mathematical model of its functionality.\n- The authors make the claim that a minimal biological neuron model must adhere to the online learning and local update rule constraints. It would be good for the authors to motivate this assumption as it is not self-evident.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The weaknesses and limitations of the work are discussed in the main review. The work would have any obvious direct negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "HQIrxLRhmzc", "reviewer": "Reviewer_up9Q", "summary": "This paper presents a normative study by combining the inspirations from fourth-order blind identification and similarity matching to derive a novel learning rule to achieve independent component analysis in a single layer recurrent network. The derived learning rule only depends on the overall neuronal responses in the network, rather than the detailed population activity pattern, which makes the learning rule more biologically plausible compared with previous work. The paper also justifies some possible neural substrates to access the information of overall neuronal activities in a neuronal population. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is based on previous work of similarity matching and marries it to fourth-order blind identification to achieve independent component analysis. The derived learning rule is novel, and the mathematical analysis is solid. The whole paper is structure-wise and well written. Overall I think it is a good normative study worth publishing.\n\nMajor concerns\n* Motivation of minimax optimization.\nWhen I read the paper I feel the flow has a jump at section 4.1 in that I cannot get the motivation of converting the original minimized objective function into a minimax objective function, and the motivation of proposing the particular form of minimized objective. Is this minimax objective a necessary step to implement the algorithm in a network? Or the previous similarity matching algorithm optimize a minimax objective function? I wish the authors could add more motivations in a revised manuscript.\n\n\nMinor concerns\n* Eq. 5: is $\\Lambda$ a parameter that needs to be optimized? Reading through the whole paper I don’t see how $\\Lambda$ is constrained.\n\n* Eq. 13: I can tell $\\alpha_t  = |y_t|^2$ resulted from a rotation matrix defined in step 2b. However, I don’t see why $\\alpha_t = |s_t|^2$. Is there an implicit assumption that the mixing matrix A (Eq. 1) is also a rotation matrix?\n\n* line 155: Eq. 10 --> Eq. 8\n\n* Eq. 18: the authors called this a thresholded nonlinearly modulated Hebbian rule. I don’t see how the thresholding process comes from. I know the $\\Lambda$ determines LTP and LTD but there seems to be no thresholding like all or none process similar with thresholding in spike generation. \n\n* The Eq. 6 and definition of $\\Gamma_x$ at line 140 can be simplified as the result appearing Eq. 12, which could simplifies the math equations a lot.\n\n* Comparing Fig. 1B and neural dynamics shown in Eq. 16, the recurrent connection weight should be $I-M$ instead of $M$ since typically there is a leak term.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dear authors,

reviewers have reached a clear positive consensus on your work and I
therefore endorse your paper for acceptance.

Best regards
The AC
Decision: Accept (Spotlight)