Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources
Authors: Bariscan Bozkurt
Cengiz Pehlevan
Alper Tunga Erdogan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Biologically Plausible Neural Networks
Blind Source Separation
Weighted Similarity Matching
Determinant Maximization

Abstract: Extraction of latent sources of complex stimuli is critical for making sense of the world. While the brain solves this blind source separation (BSS) problem continuously, its algorithms remain unknown. Previous work on biologically-plausible BSS algorithms assumed that observed signals are linear mixtures of statistically independent or uncorrelated sources, limiting the domain of applicability of these algorithms. To overcome this limitation, we propose novel biologically-plausible neural networks for the blind separation of potentially dependent/correlated sources. Differing from previous work, we assume some general geometric, not statistical, conditions on the source vectors allowing separation of potentially dependent/correlated sources. Concretely, we assume that the source vectors are sufficiently scattered in their domains which can be described by certain polytopes. Then, we consider recovery of these sources by the Det-Max criterion, which maximizes the determinant of the output correlation matrix to enforce a similar spread for the source estimates. Starting from this normative principle, and using a weighted similarity matching approach that enables arbitrary linear transformations adaptable by local learning rules, we derive two-layer biologically-plausible neural network algorithms that can separate mixtures into sources coming from a variety of source domains. We demonstrate that our algorithms outperform other biologically-plausible BSS algorithms on correlated source separation problems.

Biologically-Plausibbe Determinant Maximization Neural Networks for Blind Separation of Correlated Sources
Bariscan Bozkurt Cengiz Pehlevan Alper T. Erdogan
KUIS Center, Koe University, Turkey 'EEE Department, Koe University, Turkey 'John A. Paulson School of Engineering AApplied Sciences and Center for
Brain Science, Harvard University, Cambridge, 02138 MA, USA
(bbozkurt15, alperdogan] (ku. edu.tr cpehlevan harvard.
Abstract
Extraction of latent sources of complex stimuli is critical for making sense of the world. While the brain solves this blind source separation (BSS) problem tinuously, its algorithms remain unknown. Previous work on biologically-plassin BSS algorithms assumed that observed signals linear mixtures of statistically independent or uncorrelated sources, limiting the domain of applicability of these algorithms. To overcome this limitation, we propose novel poologccally-plasibbe neural networks for the blind separation of potentially dependdent/correlated sources. Differing from previous work, we assume some general geometric. not statistical, conditions on the source vectors allowing separation of potentially dependen//correlated sources. Concretely, we assume that the source vectors are sufficiently scattered in their domains which can be described by certain polytopes. Then, we consider recovery of these sources by the Det-Max criterion, which maximizes the determinant of the output correlation matrix to enforce a similar spread for source estimates. Starting from this normative principle, and using a weighted similarity matching approach that enables arbitrary linear transformations adaptable by local learning rules, we derive two-layer viologically-plausiii neural network algorithms that can separate mixtures into sources coming from a variety of source domains. We demonstrate that our algorithms outperform other irlogicclly-plausibbe BSS algorithms correlated source separation problems.
Introduction
Our brains constantly and effortlessly extract latent causes, or sources, of complex visual, auditory olfactory stimuli sensed by sensory organs II This extraction is mostly done without any instruction. in an unsupervised manner, making the process an instance of the blind source separation (BSS) problem 12 13.. Indeed, visual and auditory cortical receptive fields were argued be the result of performing BSS on natural images 2 and sounds []. The wide-spread use of BSS in the
brain suggests the existence of generic circuit motif's that perform this task [1]. Consequently, the literature on biologically-plause
neural network algorithms for BSS is growing [15
Because BSS is underdetermined inverse problem, BSS algorithms make generative assumptions on observations. In most instances of the biologically-plausibe BSS algorithms, complex stimuli are assumed be linear mixtures of latent sources. This assumption is particularly fruitful and used to model, for example, natural images
20) and responses of olfactory neurons to complex odorants 21 [ However, linear mixing by itself is not sufficient for source identifiability: further assump-
tions are needed. Previous work on biologically-plaue plausible algorithms for BSS of linear mixtures
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

However, these (a) B (sparse)
(b) Bi (anti- (parr-sparse)
(c) (normalized nonnegative)
(d) Be (nonnegative anti unti-sparse)
(e) Be (nonnegative sparse)
Figure 1: Examples of source domains leading to identifiable generative models.
assumed sources be statistically independent assumptions are very limiting when considering real data where sources can themselves be correlated.
- or uncorrelated T6 18
In this paper, address the limitation imposed by independence assumptions and provide biologically-plassiei BSS neural networks that can separate potentially correlated sources. We achieve this by considering various general geometric identifiability conditions on sources instead of statistical assumptions like independence or uncorrelatedness. In particular, 1) we make natural assumptions the domains of source vectors-like nonnegativity, sparsity, anti-sparsity bounded- ness (Figure D and 2) we assume that latent source vectors are sufficiently spread in their domain 25 26 Because these identifiability conditions are stochastic in nature, our neural networks able to separate both independent and dependent sources.
We derive our peglooically-plausibee algorithms from normative principle. A common method for exploiting our geometric identifiability conditions is to disperse latent vector estimates across their presumed domain by maximizing the determinant of their sample correlation matrix, i.c., the Det-Max approach [25] 27/ Starting from a Det-Max objective function with constraints that specify the domain of source vectors, and using mathematical tools introduced for mapping optimization algorithms to adaptive Hebbian neural networks [88
we derive two-layered neural networks that separate potentially correlated sources from their linear mixtures (Figure 2). These networks contain feedforward, recurrent and feedback synaptic connections updated via Hebbian anti-Hebbian update rules. The domain of latent sources determines the structure of the output layer of the neural network (Figure Z Table M and Appendix [)
31 32
In summary, main contributions in this article are the following:
We propose a normative framework for generating biologically plausible neural networks that are capable of separating correlated sources from their mixtures by deriving them from a Det-Max objective function subject to source domain constraints.
Our framework can handle infinitely many source types by exploiting their source domain topology.
We demonstrate the performance of our networks in simulations with synthetic and realistic data.
1.1 Other related work
Several algorithmss for separation of linearly mixed and correlated sources have been proposed outside the domain of biologically-plasiin BSS. These algorithms make other forms of assumptions on the latent sources. Nonnegative matrix factorization (NMF) assumes that the latent vectors are Simplex structured matrix factorization (SSMF) assumes that the latent vectors are members of the unit-simplex (25 37.. Sparse component analysis (SCA) often assumes Antisparse bounded component
nonnegative 113
5X 35
that the latent vectors lie in the unity ---orm-balll [30
38
42).
analysis (BCA) assumes latent vectors are in the -nrm--b polytopic matrix factorization (PMF) extends the identifiability-nnan polytopes obeying a particular symmetry restriction
2y 43 Recently introduced
domains to infinitely many
The mapping of optimization algorithms to biologically-plane plausible neural networks have been formal- ized in the similarity matching framework 31I 32 46 47 Several BSS algorithms were proposed
within this framework: 1) Nonnegative Similarity Matching (NSM) [16 48 separates linear mix- tures of uncorrelated nonnegative sources, 2)
19 separates independent sources, and 3) Bounded
Similarity Matching (BSM) separates uncorrelated anti-sparse bounded sources from foo ball

---

W WHX HX
Wyn
E
y1
Feedfonward Hebbian
37
W HX
WYH
y1
Recurent Arti-Heettiee
Feedback Hebbian
32
My
322
M
M
My
3m
mm
W FY
W HY
Un
(a) General source domain with sparse components
(b) Antisparse sources
Figure 2: Det-Max WSM neural network for blind source separation. The network takes a mixed input x and produces latent components y at the output. The output layer depends the choice of source domain. Mutually sparse components are connected by inhibitory neurons at the output layer.
18 BSM introduced a weighted inner product-based similarity criterion, referred to as the weighted similarity matching (WSM). Compared to these algorithm, the neural network algorithms propose in this article 1) cover more general source domains, 2) handle potentially correlated sources, 3) two-layer WSM architecture (relative single layer WSM architecture of BSM, which is not capable of generating arbitrary linear transformations) and 4) offer a general framework for neural-nettwok--basse optimization of the Det-Max criterion.
Problem statement
2.1 Sources
We assume that there are n real-valued sources, represented by the vector s E P, where P is particular subset of R" Our algorithms will address wide range of source domains. We list some examples before giving a more general criterion:
Bounded sparse sources: A natural convex domain choice for sparse sources is the unit & norm ball Bg = {s |||||| S 1} (Figure I (a)). The use of f--ormm as a convex (non)sparsity measure has been quite successful with various applications including sparse dictionary learning/component analysis 30 39
50 and modeling of V1 receptive fields (2).
Bounded anti-sparse sources: A common domain choice for anti-sparse sources is the unit 600- norm-ball: Br_ = (s |||| S }] (Figure (b)). vectors drawn from B,_. are well-spread inside this set, some samples would contain near-peak magnitude values simultaneously at all their components. The potential equal spreading of values among the components justifies the term "anti-sparse" [5T] or "democratic" [52] component representations. This choice is well-suited for both applications in natural images and digital communication constellations
28
43).
Normalized nonnegative sources: Simplex structured matrix factorization 25
36
37
uses the unit
simplex 35 53 A = {s > 0, 1's 1} (Figure (c)) the source domain. Nonnegativity of sources naturally arises in biological context, for example in demixing olfactory mixtures (54)
Nonnegative bounded anti-sparse sources: A non-degenerate polytopic choice of the nonnegative sources can be obtained through the combination of -ati-sparsessss and nonnegativity constraints. This corresponds to the intersection of B with the nonnegative orthant TR,, represented as
B Br__ R 26 (Figure (d)).
Nonnegative bounded sparse sources: Another polytopic choice for nonnegative sources can be obtained through combination of the sparsity and nonnegativity constraints which yields the intersection of B, with the nonnegative orthant R,, 26: B B, n R (Figure (e))
Except the unit simplex A, all the examples above are examples of an infinite set of identifi- able polytopes whose symmetry groups are restricted to the combinations of component per- mutations and sign alterations formalized in PMF framework for BSS 44 Further, in-
3

---

stead of homogeneous choice of features, such as sparsity and nonnegativity, globally im- posed on all elements of the component vector, we can assign these attributes at the subvector level and still obtain identifiable polytopes. For example, the reference 226 provides the set
Per SER" $1, E [[,,1]] 83 E [0, 1], $2 1
S as a simple il- 83
lustration of such polytopes with heterogeneous structure where S3 is nonnegative, $1 $2 are signed, and 81 52 1", $2 #3 1" are sparse subvectors, while sparsity is not globally imposed. this particular source domains including the unit simplex, and the subset of
article, concentrate identifiable polytopes for which the attributes such as sparsity and nonnegativity are defined at the subvector level in the general form
(1) where I+ C Z, is the index set for nonnegative sources, and I, is its complement, S.y, is the subvector constructed from the elements with indices in Tk. and L is the number of sparsity constraints imposed in the subvector level.
P= {s E TR"
E [-1, I,, 8, E 1j. I-, || 1 Ts S En. k E Z
The Det-Max criterion for BSS is based on the assumption that the source samples well-spread in their presumed domain. The references [55] and [26] provide precise conditions on the scattering of source samples which guarantee their identifiability for the unit simplex and polytopes, respectively. Appendix A provides a brief summary of these conditions.
We emphasize that our assumptions about the sources are deterministic. Therefore, our proposed algorithms not exploit any stochastic assumptions such as independence or uncorrelatedness, and can separate both independent and dependent (potentially correlated) sources.
2.2 Mixing
The sources s; are mixed through a mixing matrix A gmx=
x, Ast.
E Z.
(2)
We only consider the (over)determined case with m 2 n and assume that the mixing matrix is full- rank. While we consider noiseless mixtures to achieve perfect separability, the optimization setting proposed for the online algorithm features a particular objective function that safeguards against potential noise presence. We use Stt) Si mmxt to represent data snapshot matrices, at time f, for sources and mixtures, respectively.
Se Raxt and X(t) x1 Xt
2.3 Separation
The goal of the source separation is obtain an estimate of S(t) from the mixture measurements X(t) when the mixing matrix A is unknown. We use the notation y, to refer to source estimates, which are linear transformations of observations, i.e., y; = Wx,, where W = We define Y: Raxt as the output snapshot matrix. "Ideal separation" is defined
(t) = Yı Y2 as the condition where the outputs are scaled and permuted versions of original sources, i.e., they satisfy y: = PAss, where P is permutation matrix, and A is full rank diagonal matrix.
3 Determinant maximization based blind source separation
Among several alternative solution methods for the BSS problem, determinant-mmxxiiizztoonn (Det- Max) criterion has been proposed within the NMF, BCA, and PMF frameworks, |2 Here, the separator trained to maximize the dlttg-ntttmminant of the sample correlation matrix for the separator outputs, J(W) = (og(dct(R,,(tt))) where R. (t) is the sample correlation matrix i= y:y? (I)Y Further, during the training process, the separator outputs
28
30
35
44]
(1) = E are constrained to lie inside the presumed source domain, i.e. P. As a result, we can pose the corresponding optimization problem 126 35
maximize Y()
log(det(Y(t))UUu)
(3a)
subject to yi E 1.
(3b)

---

where we ignored the constant term. Here, the determinant of the correlation matrix acts as spread measure for the output samples. If the original source samples (S)..... S.} are sufficiently scattered
inside the source domain P. as described in Section 2.1 and Appendix A then the global solution of
this optimization can be shown achieve perfect separation 226 55
An alternative optimization formulation of determiinant-maxiiizztoonn based on weighted similarity matching
Here, we reformulate the Det-Max problem described above in way that allows derivation of palogically-plausihbee neural network for the linear BSS setup in Section 2 to all source types discusses in 2.T
Our formulation applies
We propose the following optimization problem:
minimize PL. (t)....t.)........ Di. (1). (t. mn (t). subject to
log(Di. + Σ log(D2,úx())
(4a)
1m1
X(I) x()) (()H(( 0,
(4b)
H(t) H(t) ((t))D2(t 0,
(4c)
y: y.EP, =0 1,...., n,
(4d)
Di(1) diag Dr (1), ...., Dn (f)), 1= 1,2, (t), D1,22 (t)... Dm (t) > 0, [ 1= 1,2
(4e)
(4f)
Here, X(t) E m is the matrix containing input (mixture) vectors, Y(t) E Buxt is the matrix containing output vectors, H() E Raxt is a slack variable containing an intermediate signal
(h, E R" -- I, ....,, J). corresponding to the hidden layer of the neural network implementation in Section 5 in its columns H(t) = hi ha
Di.n. (t) for I 1,2 2 are nonnegative slack variables be described below, and D; is the diagonal matrix containing weights Di,it i. for = ...... n and = 1,2. The constraint 4dh ensures that the outputs lie in the presumed domain of sources.
h, J. Dt,11(t., Dr.22 (t)..... (t)
This problem is related to the weighted similarity matching (WSM) objective introduced 18. Con- straints 4b and 4c) define two separate WSM conditions. In particular, the equality constraint in 44b a WSM constraint between inputs and the intermediate signal (t). This constraint imposes that the pairwise weighted correlations of the signal (h., 1, 1,..., J are the same as correlations among the elements of the input signal {x,, I,..., 4). i.e., x² Xj = h²" D
(1)h,, Vi,j E (I..... I).
(t), (t) ... D1.mn (f) correspond to inner product weights used in these equalities. Sim- ilarly, the equality constraint in (4c) defines a WSM constraint between the intermediate sig- nal and outputs. This equality can be written h h, y D2 i.j E (l,..., t). and D2,1 (t), (2) D2,nn (f) correspond to the inner product weights used in these equalities. The optimization involves minimizing the logarithm of the determinant of the weighting matrices.
Now we state the relation between our WSM-based objective and the original Det-Max criterion D
Theorem If Xt) is full column-rant then global optimal Yt)) solutions of and coincide.
Proof of Theorem 7 See Appendix B for the proof. The proof relies on a lemma that states that the optimization constraints enforce inputs and outputs to be related by an arbitrary linear transformation.
Biologically-plausibe neural networks for WSM-based BSS
The optimization problems we considered so far were in an offline setting, where all inputs are observed together and all outputs are produced together. However, biology operates in an online fashion, observing an input and producing the corresponding output, before seeing the next input. Therefore, in this section, we first introduce an online version of the batch WSM-problem 4 Then we show that the corresponding gradient descent algorithm leads to a two-layer neural network with belogiccally-llusibbe local update rules.
5

---

5.1 Online optimization setting for WSM-based BSS
We first propose an online extension of WSM-based BSS (4. In the online setting, past outputs cannot be altered, but past imputs and outputs still carry valuable information about solving the BSS problem. We will write down an optimization problem whose goal is to produce the sources ye given a mixture Xt, while exploiting information from all the fixed previous inputs and outputs.
We first introduce our notation. We consider exponential weighting of the signals as a recipe for dynamical adjustment potential nonstationarity in the data. We define the weighted input data snapshot matrix by time as, (t) - 7x=
7 X (()) where is the forgetting factor and (t)) = diag(?' ....,.; 1). The exponential weighting emphasizes recent mixtures by reducing the impact of past samples. Similarly, we define the corresponding weighted
output snapshot matrix for output as (t) "
(t)) and the
7
7h h, (t)) We further define T lim = Σk=o 2k = I -- as a measure of the effective time window length for sample correlation calculations based on the exponential weights.
hidden layer vectors (t)
-
In order to derive an online cost function, we first converted equality constraints in 44) and to similarity matching cost functions J1 (t), (t) = (t) (t) H1)) (t)
4c
Jz (t),,, (2),), Y(t)) = 2-2
1 H+())² (()))
D2(t) p. Then, a weighted combination
of similarity matching costs and the objective function in 4a yields the final cost function
(t(), Y(t))
AsM (H Di(t)) (1 3) (t),,, (())
+(1-AsM/ Σ
log( 1I, (t)) Σ (t)) (5)
Here, 33 [0. 1] and ASM E [0, 1] are parameters that convexly combine similarity matching costs and the objective function. Finally, can state the online optimization problem for determining the current output ye. the corresponding hidden state h, and for updating the gain parameters D,(t) for I 1, as
minimize y ,i, (0) DI (1)...... (0, (i)))()) na, (t).... D2 (t).
(H((), D2(t) Y())
(6a)
subject to
Yr P,
(6b)
Dt(t) = diag( Dt, ..... Di (t)), 1 1,2, (t), (t) Dr,m (t) > 0, = 1,2
(6c)
(6d)
As shown in Appendix
C.I part of J that depends on he and y: can be written as
2h D, Mn(DD)
4h, D (() (x
+2y7 ((
D)) + 2h? Mn()) (7)
where the dependence on past inputs and outputs appear in the weighted correlation matrices:
(t) = - Ei= (2)6-- (nx(1) 6 (y = : I (7))-
(8)
Wr = -
5.2 Description of network dynamics for bounded anti-sparse sources
We now show that the gradient-descent minimization of the online WSM cost function in 6 can be interpreted as the dynamics neural network with local learning rules. The exact network architecture is determined by the presumed identifiable source domain P, which can be chosen in infinitely many ways. In this section, we concentrate on the domain choice P = Bxx as an illustrative example. In Section 5.3 we discuss how to generalize the results of this section by modifying the output layer for different identifiable source domains. We start by writing the update expressions for the optimization variables based on the gradients of J [h; yr, D Dt()), Dz)))):

---

Update dynamics for by: Following previous work 48 56 and using the gradient of in A.8
with respect to he, we can write down an update dynamics for h, in the form
dv(r)
dr = -v(r7 1 + (D(()))
+ + (1 D2(U((T))
(9)
V=(T)
(T)(T) = σo ASMI'H (t(((((()(())))))))))))))))))))) B) +BD),(())))
for i == -..... n, (10)
where I (t) is diagonal matrix containing diagonal elements of (t) and M (t) = (t)
(t) is the clipping function, defined as dA(x) = Asign(z)
I
- < x S A, This dy-
otherwise.
namics can be shown to minimize [56] Here v(r) is an internal variable that could interpreted as the voltage dynamics of a biological neuron, and is defined based on linear transformation of he in dA.9). Equation D defines v(T) dynamics from the gradient of 77) with respect to h, (A.8). Due
to the positive definite linear map A.9), the expression in A. also serves a descent direction for v(r). Furthermore, o(-) function is the projection onto AB where [-A, A] is the presumed
dynamic range for the components of h, We note that there is explicit constraint set for h, in the online optimization setting of Section [.I] and therefore, A can be chosen as large as desired in the
actual implementation. We included the nonlinearity in 10 to model the limited dynamic range of an actual (biological) neuron.
Update dynamics for output yı: We write the update dynamics for the output y.. based on A.11 as
du(r) -u(r) + (y)(t) My (2)))))))
(11)
dr
(T(
(T) = σ1 (Tr(())2,a()))
for 1,,...n,
(12)
which is derived using the same approach for h2, where we used the descent direction expression in A. and the substitution in A. 12.. Here, [t) is diagonal matrix containing diagonal elements of MM (t) and (t = My () (t). Note that the nonlinear mapping σ (-) is the projection onto the presumed domain of sources, i.e., P Bxx. which is elementwise clipping operation.
The state space representations
IOp and network with imput x,, hidden layer activation hy output layer activation yt, W. (W Hx) and Wy (W are the feedforward (feedback) synaptic weight matrices for the first and the second layers, respectively, and Мμ and My are recurrent synaptic weight matrices for the first and the second layers, respectively. The corresponding neural network schematic is provided in Figure (b). The gain and synaptic weight dynamics below describe the learning mechanism for this network:
correspond a two-layer recurrent neural
Update dynamics for gains Di.s:: Using the derivative of the cost function with respect to D1,5 in A.13 can write the dynamics corresponding to the gain variable 11,is as
dD).ua(t) dt
P.D:
3) (.)))) || Hx.,, (1 ASM) Di,a(l)'
(13)
where μDD corresponds to the learning time-constant. Similarly, for the gain variable D2,5. the corresponding coefficient dynamics expression based on (A.14) is given by
(t) (A5M3) I. (l || (B) (1 ASM) Da.a(1)
PDD2
(14)
dl.
where MDD corresponds to the learning time-constant.
The inverses of the inner product weights Dr.ss correspond to homeostatic gain parameters. The inspection of the gain updates in (3)) and ( leads to an interesting observation: whether the corresponding gain is going to increase or decrease depends on the balance between the norms the recurrent and the feedforward synaptic strengths, which are the statistical indicators of the recent output and input activations, respectively. Hence, the homeostatic gain of the neuron will increase (decrease) if the level of recent output activations falls behind (surpasses) the level of recent input
7

---

activations to balance input/output energy levels. The resulting dynamics align with the experimental homeostatic balance observed in biological neurons (57].
Based on the definitions of the synaptic weight matrices in S we can write their updates as
(t (t + 1) = 1 (1- )h,b,, bab,
+
(t) (1 - y'v? (15) hhx?, Wy = WY
W WHx -1) 7 W (x
+
b,
These updates are local in the sense that they only depend on variables available to the synapse, and hence are biologically plausible.
5.3 Det-max WSM neural network examples for more general source domains
Det-Max Neural Network obtained for the source domain p Bx in Section 5.2 can extended to more general identifiable source domains by only changing the output dynamics. In Appendix D we provide illustrative examples for different identifiable domain choices. Table summarizes the output dynamics obtained for the identifiable source domain examples in Figure
Table I:
Example source domains from Figure and the corresponding output dynamies.
Source Domain
Outpuf Dynamics
Output Activation
P Boo Nonnegative Anti-sparse
ny(r) y.((T) at ( (rya(0Pa,5t))
p B1
(r,(7) STA,(e) Agarl DITy (110
Wilr)
Sparse
dutr] -a(r) + Exzo (x(((T) - A1(r)
(1(T) Rel.U(((T))
P = B
V(,(f) ReLU y A1(r))
(((r)
Nonnegative
da(t] -a(T) +2 VC -1x
Sparse
A1(T) ReLU(a(F))
P= A Unit Simplex
Vt.(F) ReLU AS000-0TY,,000000 (x
HalUle
day(r) -A1(F) + En 1
We can make the following observations on Table I (1) For sparse and unit simplex settings, there is additional inhibitory neuron which takes input from all outputs and whose activation is the inhibitory signal X (T). (2) The source attributes, which are globally defined over all sources, determine the activation functions the output layer. The proposed framework can be applied to any polytope described by T for which the corresponding Det-Max neural network will contain combinations of activation functions in Table as illustrated in Figure (a).
6 Numerical experiments
In this section, we illustrate the applications of the proposed WSM-based BSS framework for both synthetic and natural sources. More details on these experiments and additional examples are provided in Appendix E including sparse dictionary learning. Our implementation code is publicly available
6.1 Synthetically correlated source separation
In order illustrate the correlated source separation capability of the proposed WSM neural networks, we consider a numerical experiment with five copula-T distributed (uniform and correlated) sources. For the correlation calibration matrix for these sources, we use Toeplitz matrix whose first row is [1 P P p]. The parameter determines the correlation level, and we considered the range (0, 0.6) for this parameter. These sources are mixed with identically distributed (i.i.d.) standard normal random variables. The mixtures are corrupted by i.i.d.
random matrix with independent and
Slind-Source -Saurce-eppaatoon

---

normal noise corresponding to 30dB higallto-noise ratio (SNR) level. In this experiment, we employ the sateeegaavv--anippaae WSM neural network (Figure g in Appendix D.2) whose activation functions at the output layer are nomnegaiive-llippnng functions, as the sources are nonnegative uniform random variables.
We compared the signal-to-interferenne- plus-nois-ppower-rati (SINR) perfor- mance of our algorithm with the NSM algorithm [T6] Infomax ICA algorithm [I as implemented in Python MNE Toolbox (58), LD-InfoMax algorithm
Monnegative Anti Anti-sparse Source Separation SINR Results
35
30
[59]. and PMF algorithm [26]. Figure B shows the SINR performances of these algorithms (averaged over 300 realizations) as a function of the correlation parameter p. We observe that our WSM-based network performs well despite correlations. In contrast, performance of NSM and ICA algorithms, which assume uncorrelated sources, degrade noticeably with increas- ing correlation levels. In addition, we note that the performance of batch Det-Max algorithms, i..., LD-InfoMax and PMF, also robust against source correlations. Furthermore, due to their batch nature, these algorithms typically achieved better performance results than neural network with restieerestiiction, as expected.
20
WSM
25
10
ICA-Infomax
LD-infoMax
PMF
8.0
0.1
0.2
0.3
0.4
0.5
0.6
p
Figure 3: The SINRs of WSM, NSM, ICA, PMF, and LD-InfoMax versus the correlation parameter p.
6.2 Image separation
To further illustrate the correlated source separation advantage of our approach, we consider a natural image separation scenario. For this example, have three RGB images with sizes 324 x 432 x 3 as sources (Figure 4.(a) The sample Pearson correlation coefficients between the images are
P12 0.263, P13 0.066, P23 0.333. We use random 5 mixing matrix whose entries are drawn from i.i.d standard normal distribution. The corresponding mixtures are shown in Figure 4.(b)
(c)
(d)
Figure 4: (a) Original RGB images, (b) Mixture RGB images, (c) ICA outputs, (d) NSM outputs (using pre-whitened mixtures), (e) WSM outputs.
We applied ICA, NSM and WSM algorithms the mixtures. Figure (.(c).d).(e shows the corre- sponding outputs. High-resolution versions of all images in this example are available in Appendix E.4 in addition to the comparisons with DD-Infomax and PMF algorithms. The Infomax ICA algorithm's 's outputs have SINR level of 13.99 22d2, and this performance is perceivable as residual interference effects in the corresponding output images. The NSM algorithm achieves significantly higher SINR level of 17.45dB and the output images visually reflect this better performance. Our algorithm achieves the best SINR level of 27. 27.49dB, and the corresponding outputs closely resemble the original source images.

---

7 Discussion and Conclusion
We proposed general framework for generating biologically plausible neural networks that are capable of separating correlated sources from their linear mixtures, and demonstrated their successful correlated source separation capability through synthetic and natural sources.
Another motivation for our work is to link network structure with function. This a long standing goal of neuroscience, however examples where this link can be achieved are limited. Our work provides concrete examples where clear links between a network's architecture-i.e. number of interneurons, connections between interneurons and output neurons, nonlinearities (frequency-cureent curves)- and its function, the type of source separation or feature extraction problem the networks solves, can be established. These links may provide insights and interpretations that might generalize to real biological circuits.
Our networks suffer from the same limitations of other recurrent bhologically-plauiblee BSS networks. First, certain hyperparameters can significantly influence algorithm performance (see Appendix E.9 Especially, the inner product gains (D;+) are sensitive the combined choices of algorithm parameters, which require careful tuning. Second, the numerical experiments with our neural networks relatively slow due the recursive computations in (0) and - 12) for hidden layer and output vectors, which is common to all biologically plausible recurrent source separation networks (see Appendix F This could perhaps be addressed by early-stopping the recursive computation 60)
Acknowledgments and Disclosure of Funding
This work/research was supported by KUIS AI Center Research Award, C. Pehlevan acknowledges support from the Intel Corporation.
References
[1] Anthony Bell and Terrence J Sejnowski. An information-maxxmizzzioon approach blind separation and blind deconvolution. Neural computation, 7(6):1 129-1159, 1995.
[2] Bruno Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, (8166533) 1996.
[3] Adelbert Bronkhorst. The cocktail party phenomenon: review of research on speech intelligibility in multiple-talker conditions. Acta Acustica united with Acustica, 86(1):117-128, 2000.
[4] Michael S Lewicki. Efficient coding of natural sounds. Nature neuroscience, 5(4::356-363, 2002.
[5] Hiroki Asari, Barak A Pearlmutter, and Anthony M Zador. Sparse representations for the cocktail party problem. Journal of Neuroscience, 26(28):7477-7490 2006.
[6] Rajiv Narayan, Virginia Best, Erol Ozmeral, Elizabeth McClaine, Micheal Dent, Barbara Shinn- Cunningham, and Kamal Sen. Cortical interference effects in the cocktail party problem. Nature neuroscience, 10(12): 1601-117777 2007.
[7] Mark A Bee and Christophe Micheyl. The cocktail party problem: what is it? how can it be solved? and why should animal behaviorists study it? Journal of comparative psychology, 122 (3):235, 2008
[8] Josh H McDermott. The cocktail party problem. Current Biology, 19(22): 1027, 2009.
[9] Nima Mesgarani and Edward F Chang. Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7397):233-25) 2012.
[10] Elana Zion Golumbie, Nai Ding, Stephan Bickel, Peter Lakatos, Catherine Schevon, Guy M McKhann, Robert R Goodman, Ronald Emerson, Ashesh D Mehta, Jonathan Z Simon, al. Mechanisms underlying selective neuronal tracking attended speech at a "cocktail party". Neuron, 77(5) 991, 2013.
10

---

[11] Takuya Isomura, Kiyoshi Kotani, and Yasuhiko Jimbo. Cultured cortical neurons can perform blind source separation according the free-energy principle. PLoS Comput Biol, 11(12): e1004643, 2015.
[12] Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent component analysis and applications. Academic press, 2010.
[13] Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari. Nonnegative matrix and tensor factorizations: applications to exploratory mulfi-way data analysis and blind source separation. John Wiley & Sons, 2009.
[14] Jitendra Sharma, Alessandra Angelucci, and Mriganka Sur. Induction of visual orientation modules in auditory cortex. Nature, 404(6780)8841-8444 2000.
[15] David M. Eagleman, Olivier J-M. D. Coenen, Vladimir Mitsner, Thomas M. Bartol, Anthony J. Bell, and Terrence J. Sejnowski. Cerebellar glomeruli: Does limited extracellular calcium implement sparse encoding strategy? In Proceedings of the 8th Annual Joint Symposium Neural Computation, 2001.
[16] Cengiz Pehlevan. Sreyas Mohan, and Dmitri B Chklovskii. Blind nonnegative source separation using biological neural networks. Neural computation, 29(11): 2017.
[17] Takuya Isomura and Taro Toyoizumi. Error-gated hebbian rule: A local learning rule for principal and independent component analysis. Scientific reports, 8(1):1-11, 2018.
[18] Alper T Erdogan and Cengiz Pehlevan. Blind bounded source separation using neural networks with local learning rules. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3812-3816. IEEE, 2020.
[19] Yanis Bahroun, Dmitri Chklovskii, and Anirvan Sengupta. A normative and biologically plausible algorithm for independent component analysis. Advances in Neural Information Processing Systems, 34, 2021.
[20] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):331-3325,, 1997.
[21] Yilun Zhang and Tatyana O Sharpee. A robust feedforward model of the olfactory system. PLoS computational biology, 12(4))e 2016.
[22] Kamesh Krishnamurthy, Ann M Hermundstad, Thierry Mora, Aleksandra M Walezak, and Vijay Balasubramanian. Disorder and the neural representation of complex odors: smelling in the real world. arXiv preprint 1rXiv:177..0196 2017.
[23] Vijay Singh, Martin Tchernookov, and Vijay Balasubramanian. What the odor is not: Estimation by elimination. Physical Review E, 104(2):024415, 2021.
[24] David Lipshutz, Cengiz Pehlevan, and Dmitri B Chklovskii. Biologically plausible single-layer networks for nonnegative independent component analysis. Biological Cybernetics, pages 1-12, 2022.
[25] Chia-Hsian Lin, Wing-Kin Kin Ma, Wei-Chiang Li, Chong- Yung Chi, and ArulMurugan Am- bikapathi. Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no-pure-picn case. IEEE Transactions on Geoscience and Remote Sensing, 53 (10):5530-5546, 2015.
[26] Gokean Tatli and Alper T. Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing, 69:5431-5447, 2021. doi: 10. 13
[27] Reinhard Schachtner, Gerhard Pöppel, and Elmar Wolfgang Lang. Towards unique solutions of non-negative matrix factorization problems by determinant criterion. Digital Signal Processing, 21(4):528 534, 2011. I.

---

[28] Alper T Erdogan. A class of bounded component analysis algorithms for the separation of both independent and dependent sources. IEEE Transactions on Signal Processing, 61(22): 5730-5743, 2013.
[29] Huseyin A Inan and Alper T Erdogan. A convolutive bounded component analysis framework for potentially nonstationary independent and/or dependent sources. IEEE Transactions on Signal Processing, 63(1):18-30 2014.
[30] Eren Babatas and Alper T Erdogan. An algorithmic framework for sparse bounded component analysis. IEEE Transactions on Signal Processing, 66(19):519455255 2018.
[31] Cengiz Pehlevan and Dmitri B Chklovskii. A normative theory of adaptive dimensionality reduction in neural networks supplementary information. Proceedings of Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015.
[32] Cengiz Pehlevan and Dmitri B Chklovskii. Neuroscience-inspinn online unsupervised learning algorithms: Artificial neural networks. IEEE Signal Processing Magazine, 36(6):88-9)) 1808888-96666 2019.
[33] Ji- Ji-Cheng Chen. The nonnegative rank factorizations of nonnegative matrices. Linear Algebra and Its Applications, 62:207-217, November 1984.
[34] Pentti Paatero and Unto Tapper. Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Envirommetrics, 5(2):11--120 1994.
[35] Xiao Fu, Kejun Huang, Nicholas D Sidiropoulos, and Wing- Kin Ma. Nonnegative matrix factorization for signal and data analytics: Identifiability, algorithms, and applications. IEEE Signal Process. Mag., 36(2):59-80) 80, 2019.
[36] Tsung-Han Chan, Wing-Kin Kin Ma. ArulMurugan Ambikapathi, and Chong-Yure Chi. A simplex volume maximization framework for hyperspectral endmember extraction. IEEE Transactions on Geoscience and Remote Sensing, 49(11):41774419 2011.
[37] Chia-Hsiang Lin, Ruiyuan Wu, Wing-Kin Ma, Chong- Yung Chi, and Yue Wang. Maximum volume inscribed ellipsoid: A new simplex-structure matrix factorization framework via facet enumeration and convex optimization. SIAM Journal on Imaging Sciences, 11())1651-1699,
2018.
[38] Pando Georgiev, Fabian Theis, and Andrzej Cichocki. Sparse component analysis and blind source separation of underdetermined mixtures. IEEE transactions on neural networks, 16(4): 992-996, 2005.
[39] David L Donoho. For most large underdetermined systems of equations, the minimal I1 -o-mm near-solution approximates the sparsest vear-solution. Communications on Pure and Applied Mathematics, 59(7):907-934 2006.
[40] Christopher Rozell, Don Johnson, Richard Baraniuk, and Bruno Olshausen. Locally competi- tive algorithms for sparse approximation. In 2007 IEEE International Conference on Image Processing, volume pages IV-169. IEEE, 2007.
[41] Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010.
[42] Eren Babatas and Alper T Erdogan. Time and frequency based sparse bounded component analysis algorithms for convolutive mixtures. Signal Processing, page 107590, 2020.
[43] Sergio Cruces. Bounded component analysis of linear mixtures: A criterion of minimum convex perimeter. IEEE Transactions on Signal Processing, 58(4):2141-1154, 2010.
[44] Gokcan Tatli and Alper T. Erdogan. Generalized polytopic matrix factorization. 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), June 2021.
[45] Bariscan Bozkurt and Alper T. Erdogan. On identifiable polytope characterization for polytopic matrix factorization. 2022 IEFE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2022.
12

---

[46] Cengiz Pehlevan and Dmitri В Chklovskii. A Soobian/anti-hebbine network derived from online non- negativee matrix factorization can cluster and discover sparse features. In 2014 48th Asilomar Conference on Signals, Systems and Computers, pages 769-775. IEEE, 2014.
[47] Anirvan Sengupta, Cengiz Pehlevan, Mariano Tepper, Alexander Genkin, and Dmitri Chklovskii. Manifold-tilingg localized receptive fields are optimal comilarity-preservii neural networks. In Advances in Neural Information Processing Systems, pages 7080-7090, 2018.
[48] Cengiz Pehlevan. A spiking neural network with local learning rules derived from nonnegative similarity matching. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7958-7962. IEEE, 2019.
[49] Kenneth Kreutz-Delgado, Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te-Won Lee, and Terrence J Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349-333 396, 2003.
[50] Yuanqing Li, Andrzej Cichocki, and Shun-ichi Amari. Analysis of sparse representation and blind source separation. Neural computation, 16(6):1 193-1234, 2004.
[51] Clément Elvira, Pierre Chainais, and Nicolas Dobigeon. Bayesian antisparse coding, IEEE Transactions on Signal Processing, 65(7):1600-167,, 2016.
[52] Christoph Studer, Tom Goldstein, Wotao Yin, and Richard G Baraniuk. Democratic representa- tions. arXiv preprint arXiv: |401.3440 2014.
[53] David Donoho and Victoria Stodden. When does non-negative matrix factorization give correct decomposition into parts? Advances in neural information processing systems, 16: 1141-1148, 2003.
[54] Agnieszka Grabska-Bann Barwiiika,,,, Simon Barthelmé, Jeff Beck, Zachary Mainen, Alexandre Pouget, and Peter E Latham. A probabilistic approach to demixing odors. Nature neuroscience, 20(1):98-106, 2017.
[55] Xiao Fu, Kejun Huang, and Nicholas D Sidiropoulos. On identifiability of nonnegative matrix factorization. IEEE Signal Processing Letters, 25(3):328 332, 2018.
[56] Christopher Rozell, Don H Johnson, Richard G Baraniuk, and Bruno A Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural computation, 20(10): 2526-2563, 2008
[57] Gina G Turrigiano. Homeostatic plasticity neuronal networks: the more things change, the more they stay the same. Trends in neurosciences, 22(5):221-227, 1999.
[58] Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A. Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, and Matti Hämäläinen. MEG and EEG data analysis with MNE-Python. Frontiers in Neuroscience, 7 (267):1-13, 2013. doi: 10.3889/nins.2201300226..
[59] Alper T Erdogan. An information maximization based blind source separation approach for dependent and independent sources. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4378-4382. IEEE, 2022.
[60] Victor Minden, Cengiz Pehlevan, and Dmitri B Chklovskii. Biologically plausible online principal component analysis without recurrent neural dynamics. In 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pages (04-111 IEEE, 2018.
[61] Cengiz Pehlevan, Alexander Genkin, and Dmitri B Chklovskii. A clustering neural network model insect olfaction. In 2017 51st Asilomar Conference on Signals, Systems, and Comput- ers, pages 593-600. IEEE, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We noticeably express the contributions and scope of the paper, i.c. providing a general framework for constructing biologically plausible neural networks for separating both independent and correlated sources while covering
more generic source domains.
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We strongly believe that this work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and our article conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Section provides the underlying assumptions for our theoretical results.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide the complete proof of Theorem D in the appendix. Moreover, we include the detailed derivations of the proposed network dynamics in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Yes, we include the code in the supplemental material with tutorial Python notebooks and multiple-run simulation scripts for our experiments addition to READMF..d file.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the appendix, we provide the hyperparameter selections and training details of the experiments. We also share our code for numerical experiments. (c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] In the appendix, we provide the SINR convergence behaviors for several experiments based on multiple realizations with corresponding percentile envelopes.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] We did not discuss the computational power and memory concerns since our method does not require an advanced computing system. An individual source separation experiment can run on basic computer.
4. If you using existing assets (e.g., code, data, models) or curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We refer to 200 and link the data in E.6 for image patches
used for sparse dictionary learning.
(b) Did you mention the license of the assets? [No] We were unable to find the official license of the image patches we used for sparse dictionary learning. However, we cite
the creators of the data (20 and give the link for the image patches in E.6
(c) Did you include any
URL? [Yes] Our code is the only new asset which we include in the supplemental material.
assets either in the supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "sJ6r3hLnie9", "reviewer": "Reviewer_ffJU", "summary": "This work focuses on the BSS problem and solves it by imposing some geometrical priors on the sources via an online weighted similarity matching algorithm (WSM). \nSince WSM does not use statistical independence of the sources to recover them, sources can be recovered even if they are correlated.\nWSM is benchmarked on 2 datasets (a synthetic and a toy dataset) and is shown to yield better performance than ICA or non-negative similarity matching (NSM).", "questions": "- Hyper-parameters setting: \nThere are many hyper-parameters to set in this method.  Did you perform a sensitivity analysis to see in which range they work ? Is there a way to find default values that would work for all kind of problems ? Can you explain how you chose the values in section E.2 and E.3 ?\n\n- Experiments on synthetic data:\nWhile the experiments on synthetic data show what they are meant to show, it would have been nice to see some experiments with real systems. ICA is used in many different settings: astronomy, neuroscience, finance (see the section 7 of Hyvärinen, Aapo, and Erkki Oja. \"Independent component analysis: algorithms and applications.\" Neural networks 13.4-5 (2000): 411-430.). \nDid you try any experiments with real data ?\n\n- Missing comparison with Det-Max criterion\nAccording to your theorem 1: Problem (3) with Det-Max criterion and Problem (4) that yield the equations for WSM updates solve essentially the same problem. Therefore it would have seem natural to check whether they yield the same results in the synthetic experiments. Did you perform this comparison ? \nThe Det-Max criterion is mathematically much simpler and seem easier to optimize. Why should practitioners use WSM instead ?", "limitations": "Ethical limitations are properly discussed", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- A general framework that applies to a large set of priors on the sources\n- An online algorithm with local update rules that is therefore more biologically plausible\n\nWeaknesses:\n- Many hyper-parameters to set and no clear rules on how to set them (but the paper is transparent about this which is a good point)\n- Experiments are on synthetic data and a toy dataset that does not really correspond to any realistic problem\n- Missing comparison with Det-Max criterion\n\nI quickly reviewed the code: it lacks documentation and unit tests are missing but is overall well structured and readable. I would still advice the authors to document every public function, make unit tests, examples and set up a continuous integration so that other researchers can easily build upon their work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "j-Wrs7f8E6", "reviewer": "Reviewer_FU8C", "summary": "The paper proposes a novel neural network architecture to perform a blind source separation task, specifically addressing the case of correlated sources. The proposed architecture is shallow (2 or 3 layers) and derived in an online manner to maximize biologically plausibility. The proposed architecture is shown to solve a determinant-maximization problem, as proved in Theorem 1 (line 172), and to adequately solve two synthetically correlated source separation toy examples.", "questions": "1. What makes the proposed method well suited from correlated sources?\n\n1. Figure 3/4 only compare the proposed method with ICA and NSM, despite a larger number of related works listed in section 1.1: NMF, SSMF, SCA, BCA, PMF, BSM. Why not comparing with these other methods?\n", "limitations": "- The paper stresses the computational cost of the proposed method, but does not give concrete examples. What is the computational cost of the proposed method (including hyperparameter tuning) in the two proposed numerical experiments? What is the computational cost of ICA and NSM?", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Disclaimer: Due to my lack of expertise on the topic, I am not able to assess the soundness of the mathematical model nor the correctness of the theorem proof. I have also not read carefully the very long appendix (31 pages, 17 figures). Due to the length of the full manuscript, a journal might be a better publishing venue to benefit from more extensive reviews.*\n\n### Originality\nThe blind source separation problem tackled by the paper is a well-known problem, addressed by a wide literature of methods. The originality seems to resides in the special case of correlated sources (as opposed to e.g. ICA which specifically assumes independent sources).\n\nThe proposed approach builds upon a recent framework called weighted similarity matching (WSM) and introduced in [15]. The difference with related works is very briefly discussed lines 70-74.\n\n### Clarity\nThe paper is clearly written, but the relation to prior work is somewhat limited. A number of existing methods are listed, but the difference with the proposed method is not clearly explained. The paper would benefit from being more pedagogical about its different original contributions. \n\n### Quality\nThe numerical experiments to demonstrate the effectiveness of the model are quite limited. It would have been interesting to compare the proposed method with more methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8azOK4QpBcA", "reviewer": "Reviewer_NHdM", "summary": "This work follows a recent line of work on formulating blind source separation problems as solutions of similarity matching objective functions. This work greatly expands existing work in the domain by proposing geometric interpretation and an objective function related to the Det-Max approach. Also, the formalism allows for the derivation of a biologically-plausible and online learning algorithm. Indeed, the model can be implemented by a two-layer neural network with local learning rules. ", "questions": "My questions relate to the minor weaknesses found above: \n\nCan you propose a concise presentation of the online result showing that your model can operate in that setting? \n\nCan you explain why would biological plausible model has to solve such problems? \n\nCan you compare your model to existing algorithms that were designed to solve it, and not only bio-inspired ones? ", "limitations": "The authors have addressed the limitations of the paper in the last section.  ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors have covered a broad class of blind source separation problems. These problems generalize the well-known problems for which there existed or not biologically plausible learning algorithms. There is a broad modeling literature on BSS, and a growing one on similarity matching, and these generalizations and extensions of both is very novel. Although it is very technical the manuscript is well written and is easy fairly easy to follow. \n\nI found three minor weaknesses in this paper that can be easily addressed. \n\nOne is that the paper claims that the resulting algorithm is only, but only presents these results in the appendix. I would have appreciated that some space of the main paper be allocated to that as it is rather central to the paper. \n\nThe second one is related to the context of the problem. The paper is at the interface of signal processing, machine learning, and neuroscience, and it is a bit much to ask the reader to be well versed in all the different BSS problems covered in the paper. A bit more context for each of the problems would help understand the importance of each of these problems and why building such biologically plausible would be useful. Are natural data mixed or present in the form presented in this paper? \n\nFinally, the work mainly compares to nonnegative similarity matching and infomax, but it could be interesting to see how the model compares to existing algorithms designed to solve the problem at hand, not only biologically inspired ones. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6Nq9meKaBg", "reviewer": "Reviewer_qu9X", "summary": "The authors present a biologically-plausible algorithm for blind source separation (BSS) of correlated input sources. By applying weight similarity matching (WSM) approach to the Det-Max optimization algorithm used for BSS, the authors show that they can derive 2-layered Hebbian neural networks that are able to separate correlated sources from linear mixtures. They then compare their methods against Independent-Components Analysis (ICA) and Nonnegative Similarity Matching (NSM) methods in two tasks: one with artificial signals and another with natural images. In the task with artificial signals, the authors show how their algorithms is robust against correlation in the input sources whereas ICA and NSM degrade as measured by SINRs. The authors report results for the task of separating mixtures of natural images that are in-line with the former results, and they cherry-pick an example that illustrated the superior performance of their algorithm.      \n\n", "questions": "Please add more experiments and report SNIR comparing against algorithms that are able to separate correlated sources. ", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper offers a general framework for deriving neural-network from the Det-Max approach, which is a novel contribution. They are the first (to the best of my knowledge) to propose a biologically-plausible algorithm that separates correlated sources. Overall, seems a worth contribution in terms of novelty. In terms of quality, I think the paper is very solid, and there are no obvious errors that I can see. The only experiment that I am missing is one that compares WSM with algorithms that are able to separate correlated sources and reporting how the performance compares there. I think this could be a necessary addition to judge rating of the paper. The paper is fairly clearly written, but I would suggest to add a section that states more clearly what the contributions of the paper are. I like that the authors highlight the limitations of their approach. In terms of significance, I think the paper is relevant for understanding how brains may accomplish BSS and to investigate novel biologically-inspired algorithmic improvements that can help advance state-of-the-art methods of BSS with correlated sources.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept