Correlative Information Maximization Based Biologically Plausible Neural Networks for Correlated Source Separation
Authors: Bariscan Bozkurt
Ateş İsfendiyaroğlu
Cengiz Pehlevan
Alper Tunga Erdogan
Venue: ICLR 2023 poster (2023)
Keywords: Biologically Plausible Neural Networks
Blind Correlated Source Separation
Correlative Information Maximization

Abstract: The brain effortlessly extracts latent causes of stimuli, but how it does this at the network level remains unknown. Most prior attempts at this problem proposed neural networks that implement independent component analysis, which works under the limitation that latent elements are mutually independent. Here, we relax this limitation and propose a biologically plausible neural network that extracts correlated latent sources by exploiting information about their domains. To derive this network, we choose maximum correlative information transfer from inputs to outputs as the separation objective under the constraint that the outputs are restricted to their presumed sets. The online formulation of this optimization problem naturally leads to neural networks with local learning rules. Our framework incorporates infinitely many source domain choices and flexibly models complex latent structures. Choices of simplex or polytopic source domains result in networks with piecewise-linear activation functions. We provide numerical examples to demonstrate the superior correlated source separation capability for both synthetic and natural sources.

Published as a conference paper at ICLR 2023
CORRELATIVE INFORMATION MAXIMIZATION BASED BIOLOGICALLY PLAUSIBLE NEURAL NETWORKS FOR CORRELATED SOURCE SEPARATION
Bariscan Bozkurt Ates Isfendiyaroglu" Cengiz Pehlevan* Alper T. Erdogan
KKUIS AI Center, Koc University, Turkey
EEE Department, Koc University, Turkey
3 Uskudar American Academy
*John A. Paulson School of Engineering & Applied Sciences and Center for Brain Science,
Harvard University, Cambridge, MA 02138, USA
(bbozkurt15, alperdogan)eku. edu.tr
opehlevan@seas Saahlevvneeeas.harvrddd. edu
ABSTRACT
The brain effortlessly extracts latent causes of stimuli, but how it does this at the network level remains unknown. Most prior attempts at this problem proposed neural networks that implement independent component analysis, which works under the limitation that latent causes are mutually independent. Here, we relax this limitation and propose a biologically plausible neural network that extracts correlated latent sources by exploiting information about their domains. To derive this network, we choose the maximum correlative information transfer from inputs to outputs as the separation objective under the constraint that the output vectors are restricted to the set where the source vectors are assumed to be located. The online formulation of this optimization problem naturally leads to neural networks with local learning rules. Our framework incorporates infinitely many set choices for the source domain and flexibly models complex latent structures. Choices simplex or polytopic source domains result in networks with piecewise- linear activation functions. We provide numerical examples to demonstrate the superior correlated source separation capability for both synthetic and natural sources.
INTRODUCTION
Extraction of latent causes, or sources, of complex stimuli sensed by sensory organs is essential for survival. Due absence of any supervision in most circumstances, this extraction must be performed in an unsupervised manner, process which has been named blind source separation (BSS) (Comon & Jutten 2010 Cichocki et al. 2009).
How BSS may be achieved in visual, auditory, or olfactory cortical circuits has attracted the attention
of many researchers, e.g. (Bell & Sejnowski] 1995| Olshausen & Field 1996. Bronkhorst 2000
Lewicki| 2002 Asari et al. 2006 Narayaa et al. 2007 Bee & Micheyl 2008 McDermott 2009
Mesgarani & Chang] [0112] Golumbic al. [003 Isomura et al.] 2015). Influential papers showed
that visual and auditory cortical receptive fields could arise from performing BSS on natural scenes
(Bell & Sejnowski 1993) Olshausen Field 1996) and sounds (Lewicki) 2002). The potential
ubiquity of BSS in the brain suggests that there exists generic neural circuit motifs for BSS (Sharma
al. 2000). Motivated by these observations, here, we present set of novel biologically plausible neural network algorithms for BSS.
BSS algorithms typically derive from normative principles. The most important one is the infor- mation maximization principle, which aims to maximize the information transferred from input mixtures to separator outputs under the restriction that the outputs satisfy a specific generative as- sumption about sources. However, Shannon mutual information is a challenging choice for quanti- fying information transfer, especially for data-drives adaptive applications, due to its reliance on the joint and conditional densities of the input and output components. This challenge is eased by the independent component analysis (ICA) framework by inducing joint densities into separable forms based on the assumption of source independence (Bell & Sejnowski 1995). In particular scenarios,

---

Published as a conference paper at ICLR 2023
the mutual independence of latent causes of real observations may not be plausible assumption (Träuble et al. 2021). To address potential dependence among latent components, Erdogan (2022) recently proposed the of the second-orde statistics-bssed correlative (log- dong-determinnt)) mutual information maximization for BSS to eliminate the need for the independence assumption, allowing for correlated source separation.
In this article, we propose an online correlative information maximization- based biologically plausi- ble neural network framework (CorInfoMax) for the BSS problem. Our motivations for the proposed framework are as follows:
The correlative mutual information objective function only dependent the second-order statistics of the inputs and outputs. Therefore, its use avoids the need for costly higher-order statistics or joint pdf estimates,
The corresponding optimization equivalent to maximization correlation, or linear depen- dence, between imput and output, natural fit for the linear inverse problem,
The framework relies only on the source domain information, eliminating the need for the source independence assumption. Therefore, neural networks constructed with this framework are ca- pable of separating correlated sources. Furthermore, the CorInfoMax framework can be used to generate neural networks for infinitely many source domains corresponding to the combination of different attributes such as sparsity, nonnegativity etc..
The optimization the proposed objective inherently leads to learning with local update rules.
CorInfoMax acts as unifying framework to generate biologically plausible neural networks for various unsupervised data decomposition methods obtain structured latent representations. such
nonnegative matrix factorization (NMF) (Fu et al. (019)), sparse component analysis (SCA)
(Babatas Erdogan 2018), bounded component analysis (BCA) (Erdogan 2013 Inan & Erdo-
gan 2014) and polytopic matrix factorization (PMF) (Tatli & Erdogan 2021
W
Ap
21
Recument Aat--rnttian
By
By
2m
Pn
yn
En
(a)
(b)
Figure 1: CorInfoMax BSS neural networks for two different canonical source domain representa- tions. Jz's and yi's 's represent inputs (mixtures) and (separator) outputs respectively, W are feed- forward weights, e; 's are errors between transformed inputs and outputs, By, the inverse of output autocorrelation matrix, represents lateral weights the output. For the canonical form (a), X, 's are Lagrangian interneurons imposing source domain constraints, Ap (A.T) represents feedforward (feedback) connections between outputs and interneurons. For the canonical form (b), interneurons on the right impose sparsity constraints on the subsets of outputs.
Figure [ illustrates CorInfoMax neural networks for two different source domain representation choices, which are three-layer neural networks with piecewise linear activation functions. We note that the proposed CorInfoMax framework, beyond solving the BSS problem, be used to learn structured and potentially correlated representations from data through the maximum correlative information transfer from inputs to the choice of the structured domain the output.
2

---

Published as a conference paper at ICLR 2023
1.1 RELATED WORK AND CONTRIBUTIONS
1.1.1 BIOLOGICALIY PLAUSIBLE NEURAL NETWORKS FOR BSS
There are different methods to solve the BSS problem through neural networks with local learning rules. These methods are differentiated on the basis of the observation models they assume and the normative approach they propose. We can list biologically plausible ICA networks as an example
category, which are based on the exploitation of the presumed mutual independence of sources (Isomura & Toyoizumi, 2018; Bahroun et al. 202T, Lipshutz et al. 2022). There exist alternative approaches which exploit different properties of the data model to replace the independence assumption with a weaker one. As an example, Pehlevan et al. (2017a) uses the nonnegativeness property along with the biologically inspired similarity matching (SM) framework (Pehlevan al., 20176) to derive biologically plausible neural networks that are capable of separating uncorrelated but potentially dependent sources. Similarly, [Erdogan & Pehlevan (2020) proposes bounded similarity matching (BSM) as an alternative approach that takes advantage of the magnitude bound- edness property for uncorrelated source separation. More recently [Bozkurt al. (2022) introduced a generic biologically plausible neural network framework based on weighted similarity matching
(WSM) introduced in Erdogan & Pehlevan (2020) and maximization of the output correlation determinant criterion used in the NMF, SCA, BCA and PMF methods. This new framework exploits the domain structure of sources to generate two- three-layer biologically plausible networks that have the ability to separate potentially correlated sources. Another examplc of biologically plausible neural networks with correlated source separation capability is offered in Simsek & Erdogan (2019). which also uses the determinant maximization criterion for the separation of antisparse sources.
Our proposed framework differs significantly from Bozkurt et al. (2022): :: Bozkurt et al. (2022) uses the similarity matching criterion, which is not employed in framework, as the main tool for generating biologically plausible networks. Therefore, the resulting network structure and learning rules are completely different from BBokkrrt et al. (2022). For example, the lateral connections of the outputs in Bozkurt et al.] (2022) are based on the output autocorrelation matrix, while the lateral connections for our proposed framework are based on the inverse of the output autocorrelation matrix. Unlike our framework, neurons in Bozkurrt et al. (2022) have learnable gains. The feedforward weights of the networks in Bozkurt et al. (2022) correspond to a cross-correlation matrix between the inputs and outputs of layer, whereas, for our proposed framework, the feedforward connections correspond to the linear predictor of the output from the input. The feedback connection matrix in BBozkurt et al.] (2022) is the transpose of the feedforward matrix which is not the case our proposed framework. Compared to the approach in Simsek & Erdogan (201)), our proposed framework is derived from information -theorecc grounds, and its scope is not limited to antisparse sources, but to infinitely many different source domains.
1.1.2 INFORMATION MAXIMIZATION FOR UNSUPERVISED LEARNING
The use of Shannon's 's mutual information maximization for various unsupervised learning tasks dates back to a couple of decades. As one of the pioncering applications, we can list Linsker's 's work on self-organizing networks, which proposes maximizing mutual information between input and its latent representation as a normative approach (Linsker 1988). Under the Gaussian assumption, the corresponding objective simplifies determinant maximization for the output covariance matrix. (1922) suggested maximizing mutual information between alternative latent vectors derived from the same input source self-supervised method for learning representations. The most well-known application of the information maximization criterion to the BSS problem is the ICA-Infomax approach by Bell & Sejnowski| (1995). The corresponding algorithm maximizes the information transferred from the input to the output under the constraint that the output components are mutually independent. For potentially correlated sources, (Frdognn (2022) proposed the use of a second-order statistics-based correlative (or log-determinant) mutual information measure for the BSS problem. This approach replaces the mutual independence assumption in the ICA framework with the source domain information, enabling the separation of both independent and dependent sources. Furthermore, it provides an information-theeretti interpretation for the determinant maximization criterion used in several unsupervised structured matrix factorization frameworks such as NMF (or simplex structured matrix factorization (SSMF)) (Chan et al. 2011 Lin et al. 2015 Fu et al. 2018 2019), SCA, BCA, and PMF More recently, Ozsoy al. (022
Becker & Hinton
3

---

Published as a conference paper at ICLR 2023
proposed maximization of the correlative information among latent representations corresponding to different augmentations of the same imput as self-supervised learning method.
The current article offers online optimization formulation for the batch correlative information maximization method of Erdogan (2022) that leads to general biologically plausible neural network generation framework for the unsupervised unmixing of potentially dependent/comellate sources.
2
PRELIMINARIES
This section aims to provide background information for the CorlnfoMaxxbased neural network framework introduced in Section For this purpose, we first describe the BSS setting assumed throughout the article in Section 2.1 Then, in Section 2.2] we provide an essential summary of the batch CorInfoMax -based BSS approach introduced (Erdogan 2022).
2.1 BLIND SOURCE SEPARATION SETTING
SOURCES: We assume BSS setting with finite number of n--imensiioaa source vectors, repre- sented by the Set = {1(1), s(2). s(N) C P. where P is a particular subset of IR.. The choice of source domain P determines the identifiability of the sources from their mixtures, the properties of the individual sources and their mutual relations. Structured unsupervised matrix factorization methods are usually defined by the source/latent domain, such
i. Normalized nonnegative sources in the NMF(SSMF) framework: A {s 2 0,1" 1}. Signal processing and machine learning applications such as hyperspectral unmixing and text mining (Abdolali & Gillis 202I), (Fu et al. 2016).
ii Bounded antisparse sources the BCA framework: Bi_. {s
|||||c S 1}. Applications
include digital communication signals Erdogan (2013).
iii. Bounded sparse sources in the SCA framework: B, {s ||||| < 1). Applications: modeling
efficient representations of stimulus such as vision Olshausen Field 1997) and sound Smith & Lewicki (006))
Nonnegative bounded antiparse sources in the Comeeative-BCAArrrmeeeoo B NRR.
Applications include natural images Erdogan (2013).
Nonnegative bounded sparse sources in the nonnegative-SCA framework: Ber+ Be, 0 R
Potential applications similar to A in (i).
Note that the sets in (ii)-(v) of the above list are the special cases of (convex) polytopes. Recently, Tath Erdogan (202T) showed that infinitely many polytopes with a certain symmetry restriction enable identifiability for the BSS problem. Each identifiable polytope choice corresponds to dif-
ferent structural assumptions on the source components. A common canonical form to describe polytopes is to use the H-representation Grünbaum et al. 19677;
P= TR ( < bp),
which corresponds to the intersection of half-spaces. Alternatively, similar Bozkurt et al. (2022).
we can consider a subset of polytopes, which we refer to as feature- polytopes, defined in terms of attributes (such as non-negativity and sparseness) assigned the subsets of components:
P P= (s
E R" | E [-i E I,, s; E [0, 1jvi I+, || 1 Jr S z I z (2)
where I, C z is the set of indexes for signed sources, and I4 is its complement, $y, is the sub- vector constructed from the elements with indices Jr. and L is the number sparsity constraints imposed on the sub-vector level. In this article, we consider both polytope representations above.
We assume linear generative model, that is, the source vectors are mixed through an unknown matrix A pm x(i) mined case, that is, m 2 n and rank(A) = n. We define X = ax(1)
MIXING:
As(i), Vi 1, ..... N. where we consider the overdeter-
xrN) ].
The purpose of the source separation setting is to recover the original source ma- trix S from the mixture matrix X up to some scaling and/or permutation ambiguities, that is, the separator output vectors (y(i)) satisfy y(i Wz(i) (t), for all i 1..... NN, where E I = is the learned separator matrix, II is permutation matrix, A a full-rank diagonal matrix and (i) refers to the estimate of the source of the sample index i.
SEPARATION:
4

---

Published as a conference paper at ICLR 2023
2.2
CORRELATIVE MUTUAL INFORMATION MAXIMIZATION FOR BSS
Erdogan 2022
proposes maximizing the (correlative) information flow from the mixtures to the
separator outputs, while the outputs restricted to lie in their presumed domain P. The corre- sponding batch optimization problem given by
maximize E RTXN
1 [X, Y)
log det det(Ry Ry + (f) log det Re + cI)
(3a)
subject to
Y., E P. 1..=...... N,
(3b)
where the objective function 10 Y) is the logedeterminant (LD) mutual information be-
tween the mixture and the separator output vectors (see Appendix A.1 and Erdogan (2022) for more information), Ry is the sample autocorrelation, i.e., Ry ance, i.e., R N (IN N { matrix for the separator output vector, Re equals Ry -R-R (R + = -cI--RRya (or cross-covariance Ray X # (wll)Y") matrix between mixture and output vectors, and Rg the sample autocorrelation (or autocovariance) matrix for the mixtures. As discussed in Appendix A.1 for sufficiently small c, R. the sample autocorrelation (covariance) matrix of the eπror vector corresponding to the best linear (affine) minimum mean square eπтоr (MMSE) estimate of the separator output vector y, from the mixture vector æ. Under the assumption that the original source samples are sufficiently scattered in P. (Fu et al. 2019. Tatli & Erdogan 2021). i.e., they
YY", (or autocovari-
Ry
Ry where Ray is the sample cross-correlation, i... Rzy XYT
form a maximal LD-entropy subset of P. (Erdogan 2022). then the optimal solution of (3) recovers the original sources up to some permutation and sign ambiguities, for sufficiently small
The biologically plausible CorInfoMax BSS neural network framework proposed in this article is obtained by replacing batch optimization in (3) with its online counterpart, as described in Section
3. METHOD: BIOLOGICALLY PLAUSIBLE NEURAL NETWORKS FOR CORRELATIVE INFORMATION MAXIMIZATION
3.1 ONLINE OPTIMIZATION SETTING FOR LD-MUTUAL INFORMATION MAXIMIZATION
We start our online optimization formulation for CorInfoMax by replacing the output and error sample autocorrelation matrices in Bap with their weighted versions
k k- Se
(K) = - Cy
(() RC)
1 Se
((i)e(())
(4)
1
ch
i=1
where 0 < Sy < the forgetting factor, W is the best linear MMSE estimator matrix (to estimate y from x), and e(k) y(i) (i) is the corresponding error vector. Therefore, can define the corresponding online CorInfoMax optimization problem as
1 log det R&S (k) + +T) log det (T (k) + cT)
maximize y(k) E R
(k)) 2
(5a)
subject to EP.
(5b)
Note that the above formulation assumes knowledge of the best linear MMSE matrix W(i), whose update is formulated as a solution to an online regularized least squares problem,
maximize (i) E mx pw Iwwl)
W (1)2 + (1) - W 1)I
(6a)
3.2 DESCRIPTION OF THE NETWORK DYNAMICS FOR SPARSE SOURCES
We now show that the gradient- mudient-asceen--ee -ascent-basdd maximization of the online CorInfoMax objective in corresponds to the neural dynamics of multilayer recurrent neural network with local learning
In this article, we "correlative mutual information" and 'I.D-mutual information" interchangeably.
5

---

Published as a conference paper at ICLR 2023
rules. Furthermore, the presumed source domain P determines the output activation functions and additional inhibitory neurons. For an illustrative example, in this section. we concentrate on the sparse special case in Section 2.1 that is, P Be, We can write the corresponding Lagrangian optimization setting as minimize maximize u(A)ER"
[(y(k), (())) =
- ((()))))))))))))) - 1).
A
To derive network dynamics, we use the proximal gradient update (Parikh et al. 2014) for (kk) with (y)(k)) (y(k)), derived in Appendix B and the projected gradient descent
the expression 14.14 for V Viik
update for A(k) using (xL(y(k), X(k)) = 1 Ily(k;; v + 1)ll leading to the following iterations:
e(k:v) = y(k; v) W(k)
(8)
V ((R) 7/v Bi) (k- 1)y(k;;)) (e) (I. (1- 1)e(k;v)) (b-)))tt)
(9)
ulkuu+ + 1) = STA(K))) (v((k; v) + my V
(10)
X(k;+11 + = ReLU ((k: - ( -
1)ll)))
(11)
where B5a(K (k) and BC- (k) are inverses of R.
(A: Rk5((ks1)))) and R (k-1) respectively, Ty (k) and 7k(k)
provided in (A.11) and (A.13, E iss the iteration index, By (v) and (v( are the learning rates for the output and X, respectively, at iteration v, (eLU(-)) is the rectified linear unit, and STx(.) is
1ya S X,
the soft- tofthhhresooliing nonlinearity defined as (T)
We
y/ sign(w:) otherwise resent the values in the final iteration Viinnl, with some abuse of notation, with y(kk) ykk Fainll) and e(k) = e(k; fima))
The neural dynamic iterations in 8 define recurrent neural network, where (() represents feedforward synaptic connections from the input =<k) to the error e(k). B (k) represents feedfor- ward connections from the error e(k) to the output (k)), and B (I)) corresponds to lateral synaptic connections among the output components. Next, we examine the learning rules for this network.
Update of inverse correlation matrices Byy (k) and B (A:): We can obtain the update expressions by applying matrix inversion lemma to (R (A:) ef)- and (RSa + =l)- derived in Appendix B to obtain (A.10) and 12.
1
ck
B$y (k + 1)
y
(BSr (k) Top (k) (k)p((k) (k))),,
(12)
Ck
by
B (k + 1)
1
ct
(B (l)) (k) (1:) B (k)(()))e (k))()))e(k))) ( (b
(13)
Se
Ch
However, note that 12). and
13) violate biological plausibility, since the multiplier "y and % de- pend on all output and error components contrasting the locality. Furthermore, the update in (13) is not local since it is only a function of the feedforward signal (k) B (k)e(k) entering into out- put neurons: the update [B the synaptic connection between the output neuron and the error neuron j requires [ze] which is a signal input to the output neuron j. To modify the updates (12)
and [3 into a biologically plausible form, we make the following observations and assumptions:
R (l) + cl = cl = B (k: + 1) = 1r, which is a reasonable assumption, as we expect the error e(k) to converge near zero in the noiseless linear observation model,
If Gy is close enough to I, and the time step k is large enough, (k) is approximately 1- Sy
Sy
Therefore, we modify the update equation of B (k + 1)
12 as
1 Sy (B)())))()))))) (k))(())(()) (k)).
B (k: +
BBS (k)
(14)
Sy
Feed-forward synaptic connections W W(k): The solution of online optimization in 6 is given by
W 1) W(k) + nw (()))))
(15)

---

Published as a conference paper at ICLR 2023
ceast-mean-squress where (k) is the step size corresponding to the adaptive least- wwwm-meauuuuee squares (LMS) update based on the MMSE criterion (Sayed 2003). Algorithm 1 below summarizes the Sparse CorInfoMax output and learning dynamics:
Algorithm Sparse CorInfoMax Algorithm
Input: Streaming data (a(k) E RT) )= Output: {y(k) E &R"
1: Initialize Sy. p(1)w, W By), (1), (1)
2: for k 1, 2,2,... .... N do
run neural output dynamics until convergence:
<(k; y(k; (())a(())
ukkx+1) =STx(kw) mo ( 1)e(k;)),
1)y(k;)))
1(k:+11 ReLLU(((k;)) (x 11y(A; (1bb)))1((((;:::+++111 + 1)11))).
4: Update feedforward synapses: W(k+ W(+) (ww)))())))
5: Update lateral synapses: B (k 1) - (B (k:) BS" (K))(())(()) (A))
6:
end for
Za shows the three-layer realization of the sparse CorInfo- Max neural network based on the network dynamics expressions in (8) (T] and the approximation B (k) s 1. The first layer corresponds to the error (e(k)) neurons, and the second layer cor-
Neural Network Realizations: Figure
responds to the output (3y()) neurons with soft thresholding activation functions. If we substitute
8 and B (k + 1) = I in 19,, we obtain (B(k) v)) Mý (k)y(k;v) + 2e
where M5S (k) Ny B (k) de I. Therefore, this gradient expression and 10
correspond
to the two-layer network shown in Figure 2b
W
Fordlonmedd Hebbiiin
W
Feadback Hebitinn
272
322
By
My
2m
4m
(a)
(b)
Figure 2: Sparse CorInfoMax Network: (a) three-layer (b) two-layer. v;'s 's and y 's represent inputs (mixtures) and (separator) outputs respectively, W are feedforward weights, e''s 's in the three- layer implementation (on the left) are errors between transformed inputs and outputs, By in (a), the inverse of output autocorrelation matrix, represents lateral weights at the output. My in (b), represents the output lateral weights, which is a diagonally modified form of the inverse of output correlation. In both representations, the rightmost interneuron impose the sparsity constraint.
The neural network derivation examples for other source domains are provided in Appendix C
3.3 DESCRIPTION OF THE NETWORK DYNAMICS FOR A CANONICAL POLYTOPE REPRESENTATION
In this section, we consider the optimization problem specified in 5 for a generic polytope with H-representation in (1). We write the corresponding online optimization setting in Lagrangian form as
minimize maximize A(k)20 y(k)eR=
C(y(k), A(+)) = J(y(k)) A(k)² (Apy(k) bp), (16)

---

Published as a conference paper at ICLR 2023
which is Min-Max problem. For the recursive update dynamics of the network output and the La- grangian variable, we obtain the derivative of the objective in (16) with respect to (k) and A(k) as
V ((v)C(y();; v), A(k;D)) Tv Byy (k- 1) v) - To B (A: 1))(k; Ap)(K;)), (17) Vx(1)(((((k))) Arr (k; bp.
(18)
Recursive update dynamics for (kk and ((k): To solve the optimization problem in (6)) the pro- jected gradient updates on the (k) and (k) lead to the following neural dynamic iterations:
y(k; v + 1) (k: v) (g V (()) ((y(k;)) A(k;
(19)
(kk v + 1) ReLU (A(k, (bp - Apy(k;
(20)
where 7T(1) (v) denotes the learning rate for À at iteration v. These iterations correspond to recurrent neural network for which we can make the following observations: i) output neurons use linear activation functions since (A) is unconstrained in (16). ii) the network contains f interneurons cor- responding to the Lagrangian vector X, where f is the number of rows of Ap (16), or the number of (n fa-fa of the corresponding polytope, iii) the nonnegativity of X implies ReLU activation functions for interneurons. The neural network architecture corresponding to the neural dynamics in (19p- 20 is shown in Figure Tal. which has layer of f interneurons to impose the polytopic constraint in
The updates of W (k:) and By (I:) follow the equations provided in Section 3.2 Although the architecture in Figure Ta alllos implementation of arbitrary polytopic source domains; f can be a large number. Alternatively, is possible to consider the subset of polytopes in (2). which are described by individual properties and the relations of source components. Appendix C.5 derives
the network dynamics for this feature-bused polytope representation, and Figure [b] illustrates its particular realization. The number of interneurons in this case is equivalent to the number of sparsity constraints in (2). which can be much less than the number of faces of the polytope.
4 NUMERICAL EXPERIMENTS
In this section, we illustrate different domain selections for sources and compare the proposed Cor- InfoMax framework with existing batch algorithms and online biologically plausible neural network approaches. We demonstrate the correlated source separation capability of the proposed framework for both synthetic and natural sources. Additional experiments and details about their implementa- tions are available in Appendix D
4.1 SYNTHETICALLY CORRELATED SOURCE SEPARATION WITH ANTISPARSE SOURCES
To illustrate the correlated source separation capability of the online CorInfoMax framework for both nonnegative and signed antisparse sources, i.e. s(i) E B _ Vi and s(i) E Bi Vi. respec- tively, we consider a BSS setting with n 5 sources and m = 10 mixtures. The 5-dimensional sources are generated using the Copula-T distribution with degrees of freedom. We control the correlation level of the sources by adjusting a Toeplitz distribution parameter matrix with a first row 10 samples for
of P P P p] for p E [0, 0.8). In each realization, we generate N each source and mix them through random matrix A E IR 10x5 whose entries are drawn from an i..dd standard normal distribution. Furthermore, we use an i.i.d. white Gaussian noise (WGN) cor- responding to the signal-to-noise ratio (SNR) level of 30dB to corrupt the mixture signals. We use antisparse CorInfoMax network in Section C.I and nonnegative CorInfoMax network in Appendix C.2 for these experiments. To compare, we also performed these experiments with biologically
plausible algorithms: online BCA (Simsek & Erdogan] 2019). WSM (Bozkurt et al.) 2022). NSM
(Pehlevan et al. 2017a), BSM (Erdogan & Pehlevan 2020). and batch altgorithms: ICA-Infomax
Bell & Sejnowski 1995), LD-InfoMax (Erdogan 2022). PMF (Tatli & Erdogan| 2021).
Figure 3 shows the
ratio (SINR) versus correlation level p curves of different algorithms for nonnegative antisparse and antisparse source separation experiments.
We observe that the proposed CorInfoMax approach achieves relatively high SINR results despite increasing P in both cases. Although the WSM curve has similar characteristie, its performance falls behind that of CarInfoMax. Moreover, the LD-InfoMax and PMF algorithms typically achieve the best results, expected, due to their batch learning settings. Furthermore, the performance
8

---

Published as a conference paper at ICLR 2023
of NSM. BSM, and ICA-InfoMax degrades with increasing source correlation because these approaches assume uncorrelated or independent sources.
Monnegative Anti-sparse Source Separation SINR Results
Anti-sparse Source Separation SINK Results
30
25
SAR (d8)
20
1.5
CONHONI
RS2
NS2
10
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
(a)
(b)
Figure 3: The SINR performances of CorInfoMax (ours), LD-InfoMax, PMF, ICA-InfoMax, NSM, and BSM, averaged over 100 realizations, (y-axis) with respect to the correlation factor p (x-axis). SINR vs. p curves for (a) nonnegative antisparse (+), (b) antisparse (Bi..) source domains.
4.2 VIDEO SEPARATION
To provide visual example and illustrate a real naturally correlated source scenario, we consider the following video separation setup: videos of 10 seconds are mixed to generate 5 mixture videos. The average (across frames) and maximum Pearson correlation coefficients for these three sources average = 10.1549, (23 average = 0.3811 and /12
average 0.11997, P13 maximum 0.5173, respectively. We use a random mixing matrix A TR 5x3 with positive 0.2587, P2
are P12
maximmmm = 0.3139, 1133,,,11331101 maxiinum
p13
entries (to of generality), which is provided in Appendix D.3.3 Since the image pixels are in the set [0, 1]. we use the nonnegative antisparse CorInfoMax network to separate the original videos. The demo video (which is available in supplementary files and whose link is provided in the footnote D visually demonstrates the separation process by the proposed approach over time. The first and second rows of the demo are the source videos and 3 of the 5 mixture videos, respectively. The last row contains the source estimates obtained by the CorInfoMax network during its unsupervised learning process. We observe that the output frames become visually better as time progresses and start to represent individual sources, In the end, the CarInfoMax network trained a stage of near-perfect separation, with peak signal-to-noise ratio (PSNR) levels of 35.60dB, 48.07dB, and 44.58dB for each source, respectively. Further details for this experiment can be found the Appendix D.3.3
nonnegative mixtures that they can be displayed proper images without loss
CONCLUSION
In this article, we propose an information-theoret framework for generating biologically plausible neural networks that are capable of separating both independent and correlated sources. The pro- posed CorInfoMax framework can be applied to infinitely many source domains, enabling a diverse set of source characterizations. In addition to solving unsupervised linear inverse problems, Corln- foMax networks have the potential to generate structured embeddings from observations based on the choice of source domains. In fact, as a future extension, consider representation frameworks that learn desirable source domain representations by adapting the output-interneuron connections in Figure Ta Finally, the proposed unsupervised framework and its potential supervised extensions can be useful for neuromorphic systems that are bound to use local learning rules.
In terms of limitations, we can list the computational complexity for simulating such networks in conventional computers, mainly due to the loop-based recurrent output computation. However, as described in Appendix D.7 the neural networks generated by the proposed framework have compu- tational loads similar to the existing biologically plausible BSS neural networks.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY
To ensure the reproducibility of our results, we provide
Detailed mathematical description the algorithms for different source domains and their neural
network implementations in Section 3.2 Appendix C.1 Appendix
C.2
Appendix
C33
Appendix
a.4 Appendix C.5
ii. Detailed information on the simulation settings of the experiments in Section in the main article,
and Appendix [
ii.. Full list of hyperparameter sets used in these experiments in Table Table in Appendix [D.4
Iv. Ablation studies on hyperparameters Appendix[D3.
V. Algorithm descriptions for special source domains in pscudo-code format in Appendix D.I
vi. Python scripts and notebooks for individual experiments to replicate the reported results in the supplementary zip file as well as in https//gittuhbccom//aiiiaaaaaau www.sii-aaaaaasl-oooooooooo
ETHICS STATEMENT
Related the algorithmic framework we propose in this article, we see no immediate ethical con- cerns. In addition, the datasets that we use have no known or reported ethical issues, to the best of our knowledge.
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This work was supported by KUIS AI Center Research Award. CP was supported by an NSF Award (DMS-2134157) and the Intel Corporation through the Intel Neuromorphic Research Community.
REFERENCES
Maryam Abdolali and Nicolas Gillis. Simplex-structured matrix factorization: Sparsity-based identifiability and provably correct algorithms. SIAM Journal on Mathematics of Data Sci-
ence, 3(2):593-623, 2021. doi: 10.1137/20M1354982. URLhttps::/ http:///....rg/111133311313
20M1354983
Hiroki Asari, Barak Pearlmutter, and Anthony Zador. Sparse representations for the cocktail party problem. Journal of Neuroscience, 26(28);7477-7490, 2006.
Eren Babatas and Alper T Erdogan. An algorithmic framework for sparse bounded component analysis. IEEE Transactions on Signal Processing, 66(19):5194-5205, August 2018.
Yanis Bahroun, Dmitri Chklovskii, and Anirvan Sengupta. A normative and biologically plausi- ble algorithm for independent component analysis. Advances in Neural Information Processing Systems, 34:7368-7384 2021.
Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):161-163, 1992.
Mark Bee and Christophe Micheyl. The cocktail party problem: what is it? how can it be solved? and why should animal behaviorists study it? Journal of comparative psychology, 122(3):235, 2008.
Anthony J Bell and Terrence J Sejnowski. An information-maximizztionn approach to blind separa- tion and blind deconvolution. Neural computation, 7(6):122--1199 1995.
Bariscan Bozkurt and Alper T Erdogan. On identifiable polytope characterization for polytopic matrix factorization. 2022 IEEE International Conference on Acoustics, Speech, and Signal Pro- cessing (ICASSP), May 2022.
10

---

Published as a conference paper at ICLR 2023
Bariscan Bozkurt, Cengiz Pehlevan, and Alper T Erdogan. Biologically-plausin determinant max- imization neural networks for blind separation of correlated sources. In Alice H. Oh, Alekh Agar- wal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
net / forum? Saaeespx_AArr44
Adelbert W Bronkhorst. The cocktail party phenomenon: A review of research on speech intelligi- bility in multiple-talker conditions. Acta Acustica united with Acustica, 86(1):117-125 2000.
Tsung-Han Han Chan, Wing-Kin Ma, ArulMurugan Ambikapathi, and Chong-Yung Chi. A simplex volume maximization framework for hyperspectral endmember extraction. IEEE Transactions on Geoscience and Remote Sensing, 49(11): 4193, May 2011.
Zizhong Chen and Jack Dongarra. Condition numbers of gaussian random matrices, SIAM Journal on Matrix Analysis and Applications, 27(3):603-62) 2005.
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari. Nonnegative matrix and tensor fuctorizations: applications to exploratory multi-way data analysis and blind source sepa- ration. John Wiley & Sons, 2009.
Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent component analysis and applications. Academic press, 2010.
Sergio Cruces. Bounded component analysis of linear mixtures: A criterion of minimum convex perimeter. IEEE Transactions Signal Processing, 58(4))21412215,, 2010.
Alper T Erdogan. A class of bounded component analysis algorithms for the separation of both independent and dependent sources. IEEE Transactions on Signal Processing, 61(22):5730-5743, August 2013.
Alper Erdogan. An information maximization based blind source separation approach for depen- dent and independent sources. In ICASSP 2022- 2022 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp. 4378 4382, 2022. doi: 10. 10.1109/CASSS443392... 2022.9746099.
Alper T Erdogan and Cengiz Pehlevan. Blind bounded source separation using neural networks with local learning rules. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3812-3816, 2020. doi: 10.1109/CCSSPP40776 2020.9053114.
Xiao Fu, Kejun Huang, Bo Yang, and N.D. Sidiropoulos. Robust volume minimization based matrix factorization for remote sensing and document clustering. IEEE Transactions Signal Process- ing, 64, 08 2016. doi: 10.1 20T.SSSP.00...662800
Xiao Fu, Kejun Huang, and Nicholas D Sidiropoulos. On identifiability of nonnegative matrix factorization. IEEE Signal Processing Letters, 25(3):328-332, January 2018.
Xiao Fu, Kejun Huang, Nicholas D Sidiropoulos, and Wing-Kin Ma. Nonnegative matrix factor- ization for signal and data analytics: Identifiability, algorithms, and applications. IEEE Signal Process. Mag., 36(2):59-85) 800, March 2019.
Elana M Zion Golumbic, Nai Ding, Stephan Bickel, Peter Lakatos, Catherine A Schevon, Guy M McKhann, Robert R Goodman, Ronald Emerson, Ashesh D Mehta, Jonathan Z Simon, et al. Mechanisms underlying selective neuronal tracking of attended speech at "cocktail party". Neu- ron, 7755)9800-91 2013.
Branko Grünbaum, Victor Klee, Micha A Perles, and Geoffrey Colin Shephard. Convex polytopes, volume 16. Springer, 1967.
Huseyin A Inan and Alper T Erdogan. Convolutive bounded component analysis algorithms for independent and dependent source separation. IEEE transactions neural networks and learning systems, 26(4))697-708, 2014.
Takuya Isomura and Taro Toyoizumi. Error-gated hebbian rule: A local learning rule for principal and independent component analysis. Scientific reports, 8(1):1111, 2018.
11

---

Published as a conference paper at ICLR 2023
Takuya Isomura, Kiyoshi Kotani, and Yasuhiko Jimbo. Cultured cortical neurons perform blind source separation according to the free-energy principle. PLoS Comput Biol, 11(12))e1004643, 2015.
Thomas Kailath, Ali Sayed, and Babak Hassibi. Linear estimation. Prentice-Hoe Hall information and system sciences series. Prentice Hall, 2000. ISBN 9780130224644.
Michael S Lewicki. Efficient coding of natural sounds. Nature neuroscience, 5(4):356-363, 2002.
Chia-Hsiang Lin, Wing-Kin Ma, Wei-Chiang Li, Chong- Yung Chi, and ArulMurugan Ambikapathi. Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no-pure-pixel case. IEEE Transactions on Geoscience and Remote Sensing, 53(10):5530- 5546, May 2015.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
David Lipshutz, Cengiz Pehlevan, and Dmitri B Chklovskii. Biologically plausible single-layer networks for nonnegative independent component analysis. Biological Cybernetics, pp. 1-12, 2022.
Josh McDermott. The cocktail party problem. Current Biology,
2009.
Nima Mesgarani and Edward F Chang. Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7399):233-236, 2012.
Rajiv Narayan, Virginia Best, Erol Ozmeral, Elizabeth McClaine, Micheal Dent, Barbara Shinn- Cunningham, and Kamal Sen. Cortical interference effects in the cocktail party problem. Nature neuroscience, 10(12):1001-160) 2007.
Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 38(66)8))60076609 1996.
Bruno Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision research, 37(23)3311-3325, 1997.
Serdar Ozsoy, Shadi Hamdan, Serean O Arik, Deniz Yuret, and Alper T Erdogan. Self-supervised learning with an information maximization criterion. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL
net wwt./orunf forum? id 5MgZAu2NR
Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends@ in Optimization, 1(3):127-239, 2014.
Cengiz Pehlevan, Sreyas Mohan, and Dmitri Chklovskii. Blind nonnegative source separation using biological neural networks. Newral computation, 29(11):9255-254, 2017a.
Cengiz Pehlevan, Anirvan Sengupta, and Dmitri Chklovskii. objectives lead to heebian/anti-hebbian networks? Neural computation, 30(1):84-124 2017b.
Why do similarity matching
Ali H Sayed. Fundamentals of filtering. John Wiley & Sons, 2003.
Jitendra Sharma, Alessandra Angelucci, and Mriganka Sur. Induction of visual orientation modules in auditory cortex. Nature, 404(6780):841-888 847, 2000.
Berfin Simsek and Alper T Erdogan. Online bounded component analysis: A simple recurrent neural network with local update rule for unsupervised separation of dependent and independent sources. In 20/9 53rd Asilomar Conference on Signals, Systems, and Computers, pp. 1639-1643, 2019. doi: 10. 101110EEEECN 90A.OCOTEAA44442211999999933 201..4449.119999811111 9044916.
Evan C. Smith and Michael S Lewicki. Efficient auditory coding. Nature, 439(7079)998--95)) 2006.
Gokean Tatli and Alper T Erdogan. Polytopic matrix factorization: Determinant maximization based criterion and identifiability. IEEE Transactions on Signal Processing, 69:5431-5447, 2021. doi:
10.1109/TSP.221.33 12918.
12

---

Published as a conference paper at ICLR 2023
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf. and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference on Machine Learning, pp. 10401-10412. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 INFORMATION THEORETIC DEFINITIONS
In this section, we review the logarithm-deterninn (LD) entropy measure and mutual information for the BSS setting introduced in Section 2.1 based on Erdogan (2022). Note that in this article, we refer LD-mutual information mynonymously as correlative mutual information. For finite set of
vectors X
{ax(1), (2),. x(N)) c RT with sample covariance matrix R2 = NxXXx xx
1 X111X"1, where X is defined as X (1) xx() x(N) ]. the deterministic LD- N2
entropy is defined in Erdogan (2022) as
H ((X) LD log det + el)
m
log (2Te)
(A.1)
2
where is small number to keep the expression away from -00. If the sample covariance R.
in this expression replaced with the true covariance, then H H (0) coincides with the Shannon differential entropy for Gaussian vector Moreover, deterministic joint LD-entropy of two sets of vectors X C R" and Y C R" can be defined as
H(X, Y (e) log det R x
+
+ m + n
log det(2re)
LD
2
2
y
log det (i
(RR
el
Ray
1)
m + n
log det (2Te)
Ryye
Ry + el
2
log (det(Res + eT) det Ry + +II
R
RR eI) Ryy)
2
m + n log (2Te)
+ 2
log det Rie + +e)
log log(2 log det + el) + log (2TT))
H(X)
L.D H(Y 1
(c)
(A.2)
L.
where R. Ry R Rz eI) -
Ryas and Ray N XXT R2 In A.2 the nota-
tion X12X is used to signify H(Y 1
(c) H( as the latter requires the of Ryjn
instead of R.. Moreover, H HY),,
x) ILD corresponds to the log-determinant of the error sample covariance of the best linear minimum mean squared estimate (MMSE) of y from æ. To verify that in the zero-mean and noiseless case, consider the MMSE estimate ý Wæ for which the Rny R- (Kailath et al. 2000). Then Rg = Elyg") = 2 Rmyl = R R_
solution is given by W = Ry 1e11RRR RRa. Therefore, if the error is defined as y ÿ. its B[R² 1 xxx" R
covariance matrix be found as desired, i.e., R. RR,, R, Ry R R Rey.
The LD-mutual information for X and Y can be defined based on the equations (A.I) and A.2 as 7(0) (X,Y) = #1 (Y) H X)= = 1 H (
1
1
log det Roy + e
log det (Ry, RTTy( (Poz + +II)
Ryy + +el)+ +
2
2
1
log del Re, + +el)
2 log (fe - Rr (Ry + el)
Ray + + C, (A.3)
2
where C (2me)
m
+ n
1oge2 a constant.
2
2
B GRADIENT DERIVATIONS FOR ONLINE OPTIMIZATION OBJECTIVE
Assuming that the mapping W(k) changes slowly over time, the current output y(k) can be implic- itly defined by the projected gradient ascent with neural dynamics. To derive the corresponding neu- ral dynamics for the output (k), need to calculate the gradient of the objective 5a with respect
14

---

Published as a conference paper at ICLR 2023
to (k). First, consider the derivative of both log det (Riy (le + eT) and log det (R (k) + eI)
with respect to (kk).
alog det (Ry (k) + (l) Tr (((n) -TV
aRy (k)
log det ( (k) + el)
(y:(k)
By. (k)
where (e(k) is the i-th element of the vector ykk), and
(R (x)+e) log det R (k) + cI) (R. (k) + tc)
(A.4)
RR (R)
Gy (u(()) (()))
(A.5)
(k) 1 - -G
In A.5 (8) denotes the standard basis vector with a at position and should not be confused with the error vector e(k). Combining (A.4) and (A.5). we obtain the following result:
llog det (Ry (k) el)
Cu ( (TeS (k) + +el) +
(y:(k)
1
y
which leads to
(() log det Ry (R)) + cI) = =2 c
1
R (R:) + cI) y()).
(A.6)
If we apply the same procedure to obtain the gradient of log det RR (k) + el) with respect to e(k), we obtain
V e((k) log det (R (k) + ef) = - ck
1 - Ee (R (k:) + -ee))
Using the composition rule, we can obtain the gradient of log det R& (k)) +e.) with respect to y(k) as follows:
V (R)) log det(R, R (k) + cI) = Oe(k)
V a(()) log det R%= + cI)
dy(kk
1 Se
Rfc (Ik) + +I) -c()))
(A.7)
ch
Finally, combining the results from (A.6) and A.7 we obtain the derivative of the objective function TTy(k)) with respect to ykk
1 V M(A) log det (R cI)
V (((()(((()))
= V v() log det det((R(t Rsy (k) + cI)
2
2
= Ey R (k) (e)
La RR (k) + (e))
(A.8)
For further simplification of (A.8). we define the recursions for Rbe (k) and (k)) based on the recursive definitions of the corresponding correlation matrices. Based on the definition in (4) can write
-1
k-1
Su c
+ Cy el
R'y (k) +el =
y
R (k- - 1)+ el) + (k())(())
1
kk
1 cB
~
y
1-1
Cy R (k: - 1) + +T) +
Su
((k))(())
(A.9)
1 cp
15

---

Published as a conference paper at ICLR 2023
Using the assumption in (A.9), take the inverse of both sides and apply the matrix inversion lemma (similar to its use in the derivation of the RLS algorithm Kailath et al. (2000)) to obtain
1 5 (R 1)) +
(RS) (k) + =f)= "y c
(k) (k 1) + ( -1) -))+
(A.10)
where
( Su c + ((a)
(k)
(R) (lc +1)=
(A.11)
1y cy
We apply the same procedure to obtain the inverse of (R(((K) + +el)
(k) + +-))==== 1 - /k
(1 (A: - 1) + +
5e ch
7a(k) (12 (k- 1) + +1 ((k))))))))) (1C (k)
1))+ +
(A.12)
where
Se Ce -1- Ee
Ye(k)
+ e(k) (R (k- - (
(A.13)
Note that plugging AA10)) into the first part of (.8) yields the following simplification:
1- Cu (R(()) + el)
1 - Sy 1 - ck
y(k)
(iR(k 1) + +eI)
c
1- 22 Eu c
+cT) 7o (h)) (P(k - + +eI) --)+e))--y(()pppp ()))))))))()))(((())))))) ()))))) b(k)
Sy
(ii(k: 1) +el)
Ey c3
(ii(k: 1) + el) (((
+ [kk (Pikk - 1) + 21(6
Sy -5
1-6y
So - [P(K -1)
1 - Sy
5y ck
c
u(k)²* (12(K 1)
I-
Cy
(R(l:: 1) + -
[ cI)
-)) y(k)
Ey - c7
---
I-5, +y(k)
Gy
(Rik 1 1)
Cy 25
T- + (k) (Rk 11
(k) Rsy (k + yy()
similar simplification can be obtained for A.12). and incorporating these simplifications into A.8 yields
V ((an)((((k)) = (k) (k)y(k) - 7/c (k) BB (k))(k))))
(A.14)
where we denote (Rty (k) + eI and (R (k) + eI by BBy (1: + 1) and BS- (k + 1) for simplicity, respectively.
16

---

Published as a conference paper at ICLR 2023
C SUPPLEMENTARY ON THE NETWORK STRUCTURES FOR THE EXAMPLE DOMAINS
We can generalize the procedure for obtaining CorInfoMax BSS networks in Section 3.2 for other source domains. The choice of source domain would affect the structure of the output layer and the potential inclusion of additional interneurons. Table summarizes the output dynamics for special source domains provided in Section 2.1 for which the derivations are provided in the following subsections.
Table I: Example source domains and the corresponding CorInfoMax network dynamics.
Source Domain
Output Dynamics
Output Activation
p
A
(oco)((((kk)))) ny (P 1)u(k;1) - B. ())(k;)) u(k;+ ReLU (y(k: +
A(v)),
A(G++ X(k: - 22
(E (1
Biuge
V ((ce))(r(();>))) - 7o m3 (A (2 Bi 1)((k;)),
v(+++1)==
Bisort +
V (((c))((( ( (M)))
((
M(K++ 1) (M (o)
P Bixt
"n 1)b(k:() -i 1)e(k;
v(k;+ = ReLU (g) my(u) v(c))
Mlk ReLU A(k:2) ma(w) (Ein (((K+++)))))
C.I DESCRIPTION THE NETWORK DYNAMICS FOR ANTISPARSE SOURCES
We consider the source domain P = B, which corresponds to antisparse sources. Similar to the
sparse CorInfoMax example in Section 3.2 we derive the network corresponding to the antisparse CorInfoMax network through the projected gradient ascent method. Since projection onto the P By is an elementwise clipping operation, unlike the sparse CorInfoMax case, we do not require any interneurons related the projection operation.
Recursive update dynamics for yk)) Based on the gradient of (5a) with respect to y(k) in A 1414
derived in Appendix B we can write the corresponding projected gradient ascent iterations for (5)
e(k;v) y(k; v) W (k))
V V v)) Try By (k))((k;)) 7a B (k))(k;)), u(k; 1) σ1 (v (y(1) y(a) I [zy(k;
where B (k) and (k) are inverses of Rt (k- - 1) and R$= (k: 1) respectively, v E the index of neural dynamic iterations, (.) is the projection onto the selected domain Bi.., which is yi sign(
-1 S yı < otherwise.
the clementwise clipping function defined as (1(1): =
The corresponding realization of the neural network is shown Figure
C.2 DESCRIPTION OF THE NETWORK DYNAMICS FOR NONNEGATIVE ANTISPARSE SOURCES
+. The treatment for this case is almost the same as the signed antisparse case in Appendix [.I] The only difference is that the projection to the source domain is performed by applying elementwise nonnegative clipping function to individual outputs.
We consider the source domain B B.
17

---

Published as a conference paper at ICLR 2023
61
y/1
W
Feedforward Hebbian
Recurrent Ant Hee-ben
01
Feadback Hebolan
02
By
Tm
Yn
en
Figure 4: Two-Layer antisparse CorInfoMax Network. I;'s and y/ 's represent inputs (mixtures) and (separator) outputs, respectively, W represents feedforward weights, e, are errors between transformed imputs and outputs, By, the inverse of output autocorrelation matrix, represents lateral weights the output. The output nonlinearities are clipping functions.
Therefore, we can write the neural network dynamics for the nonnegative antisparse case as
e(k;v) v(k;v) - (k))a((k)
Vonn)(y((k v)) Tv By (k) re (w)e((kk) y(k; v + = a (y(k; v) + mn VWN))
where the nonnegative clipping function is defined
0
y 0
(v)
1/1
< 94 S 1,
y:
1.
The corresponding neural network realization is shown in Figure
C1
V/
W
Feedforward Hebbian
Recurennt Ardi-HHebbbnn
01
Feedback Hebbian
Ø2
By
Fm
Y/
en
Figure 5: Two-Layer nonnegative antisparse CorInfoMax Network. x;'s and y 's represent inputs (mixtures) and (separator) outputs, respectively, W represents feedforward weights, e,'s are emros between transformed inputs and outputs, By. the inverse of output autocorrelation matrix, repre- sents lateral weights the output. The output nonlinearities nonnegative clipping functions.
C.3 DESCRIPTION OF THE NETWORK DYNAMICS FOR NONNEGATIVE SPARSE SOURCES
For the nonnegative sparse CorInfoMax network in Section 3.2 the only change compared to its sparse counterpart is the replacement of the soft- tettthhresholdin activation functions the output layer with the rectified linear unit. Accordingly, we can state the dynamics of output and inhibitory neurons
18

---

Published as a conference paper at ICLR 2023
(()) = -t- - BB 1))(k;))
+(ku++1)- - ReLU (z((k; + Vn )(k:
(xxt)(y((;)))) - Σ [e(k;
A(k: v + 1) = ReLU ((k; nx VM v))
The corresponding network structure is illustrated in Figure
g
y1
W
Foodforward Hobbian
Recurrent Ant Hebblun
Feadback Heboian
22
+
By
Tm
yn
Cn
Figure 6: Three-Layer nonnegative sparse CorInfoMax Network. xt's and y:'s represent inputs (mixtures) and (separator) outputs respectively, W represents feedforward weights, e; are errors between transformed imputs and outputs, By, the inverse of output autocorrelation matrix, repre- sents lateral weights at the output. The output nonlinearities are ReLU functions. The leftmost interneuron imposes sparsity constraints on the outputs through inhibition.
C.4 DESCRIPTION OF THE NETWORK DYNAMICS FOR UNIT SIMPLEX SOURCES
For the BSS setting for the unit simplex set A, which is used for nonnegative matrix factorization, we consider the following optimization problem,
maximize ((b) E R subject to | 1, 1, s(k)
[yy(k)
(A.15a)
(A.15b)
for which the Lagrangian-based Min-Max problem can be stated as
(((()) T(y(k)) - X(k)
minimize maximize v(k)ER"
In this context, contrary to the optimization problem defined in we do not require that the Lagrangian variable be nonnegative, due the equality constraint in AA. Hence, we write the network dynamics for a simplex source as
V Vu(k)[(((k;)) Jr B (k= 1)b(k; 7a BS= 1)e(k;)), v(bu+11 + ReLU (y(k: v) + Von) -((kk)))))((())))) A(k;
V [()kk(((k:))) =1
Σ Vi (1(k:v+
A(kiv + =)= 2(k; V
19

---

Published as a conference paper at ICLR 2023
Figure| demonstrates the network structure for simplex sources, which is identical to nonnegative sparse CorlnfoMax network except that the linear activation replaces the ReLU activation of the inhibitory neuron.
e1
2/1
W
Feedforward
Recurrent
01
Feedback Heboion
222
By
mm
yn
En
Figure 7: Three-Layer CorlnfoMax Network for unit simplex sources. x;'s and yı's represent in- puts (mixtures) and (separator) outputs respectively, W represents feedforward weights, e are errors between transformed inputs and outputs, By, the inverse of output autocorrelation matrix, represents lateral weights at the output. The output nonlincaritics are ReLU functions. The leftmost interneuron imposes sparsity constraints the outputs through inhibition.
C.5 DESCRIPTION THE NETWORK DYNAMICS FOR FRATURE BASED SPECIFIED POLYTOPES
In this section, we consider the source separation setting where source samples are from polytope represented in the form of (2). We expand the derivation in Bozkurt et al. (2022) (see Appendix D.6
the reference) to obtain neural network solution to the BSS problem for any identifiable polytope that can be expressed in the form of 2 Accordingly, consider the following optimization problem
maximize y(k) IR" subject to
[(y(k))
(A.16a)
1 < yk)z, 1,
(A.16b)
y(k),, 1,
(A.16c)
(k) $1 vi I....,L
(A.16d)
We write the online optimization setting in a Lagrangian Min-Max setting as follows:
(8)) (A())))))))
minimize (x)20
maximize y(k) E 1
[vy(k)) ((H)(l1(()))lll 1).
x (k)z,
0 x 0k),, 1
The proximal operator corresponding to the Lagrangian term can be written as
x argmin
2 G1y
al13+
X
(A.17)
st.. az
0
1=1
Let q" be the output of the proximal operator defined in A.17). From the first order optimality condition,
20

---

Published as a conference paper at ICLR 2023
If I+, then oj リン +
Σ
(1)) = 0. Therefore, 9j y/ Aisign(yz).
Σ
IEJi
IEJ1 s4. JEJ
st. JEJ.
j I+, then of = Σ A
IEJT s.. JEJI
As a result, defining Ia = (NiTi) as the set of dimension indices which do not appear in the sparsity constraints, we can write the corresponding output dynamics
V (((kit() - B, 1)y(k; Te Be v)) g(k:++) y(k; + V (()))((((
(A.18)
11(ku+ (k.)) (k. (Vg u 1)) where aj [)(k;))=
E A((k; Vj VjEInnz
LEJI
s.t. JEJ
Ms(k; ReLU 0](k;++1)-
Σ A((k:))
Vj E E I
LEJi s.L. JEN
([(:v+1)=0000(( I nI
Vj E I- nIa
For inhibitory neurons corresponding Lagrangian variables A1...., dL, obtain the update dynamics based on the derivative of (yy(k; v), A1 v)..... (k: as
dC(yy(k), (k), (k))
= - (1yy(k; u 11)os 1l,
dAi(k)
Ar(ksw)
11)= Ae(k;; v)
- (v) dl( A1 (k)..... Ax(k))
ddr(k)
Ai(ks))
Akk, P + 1) = ReLU (((k:)+)))
In Appendix D.2.4, we demonstrate an example setting which the underlying domain is defined
as
$1, $2 84
E
[-1, 11, 53, 95
E
0, 1].
Pez
8 SER
$1
82
$2
1,
83
5
$5
84
We summarize the neural dynamics for this specific example:
(k; (Tx,k (ki 1)),
vk ST, (TA,
+));
Va(k; + 1) = ReLU Do (k; + 1) \2(k:
u4(k; + 1) STaa(kw))
1)),
(k 1) = ReLU ReLU (k; + + 1) - (k;
11 1) = \1; nx, (v) (1 - 111 (k; v + 1)1 - 122 v + 1)] - 1s(k;; v + 1)),
12 1) >2(k; 7A, (v) 122(3; v + 1)] va + +1) (k; 1)
where okk; v) is defined as in A.18).
D SUPPLEMENTARY ON NUMERICAL EXPERIMENTS
In this section, we provide more details on the algorithmic view of the proposed approach and the numerical experiments presented. In addition, provide more examples.
21

---

Published as a conference paper at ICLR 2023
D.1 ONLINE CORINFOMAX ALGORITHM IMPLEMENTATIONS FOR SPECIAL SOURCE DOMAINS
Algorithm 2 summarizes the dynamics of the CorInfoMax network and learning rules. For each of the domain choices, the recurrent and feedforward weight updates follow (14) and [1]. respectively, and the learning step is indicated in the 4th and 5th lines in the pseudo-code. The line 3 expresses the recursive neural dynamics to obtain the output the network, and its implementation differs for different domain choices. Based the derivations in Section and Appendix C Algorithm andmmsuuareee the neural dynamic iterations for some example domains. For example, Algonithm 5 indicates the procedure obtaín the output y(k) (k) of the antisparse CorInfoMax network time step & corresponding to the mixture vector æ(k). As it is an optimization process, we introduce two variables for implementation in digital hardware: 1) numerical convergence tolerance <,, and 2) maximum number of (neural dynamic) iterations Vinax- We run the proposed neural dynamic iterations until either a convergence happens, i.e., [lyr(k; v) - y(k; v ()/l////(k;)) v)]l EG. or the loop counter reaches predetermined maximum number of iterations, that is, 1/mxx Differently from the antisparse and nonnegative antisparse networks, the other CorInfoMax neural networks include additional inhibitory neurons due to the Lagrangian Min-Max settings, and the activation of these neurons are coupled with the network's output. Therefore, inhibitory neurons are updated in neural dynamics based on the gradient of the Lagrangian objective.
Algorithm 2 Online CorInfoMax pseudo-code
Input: A streaming data of {æ(k) E [R 1k=1"
Output: {3(K) E RT 1 Jkml
1: Initialize Sy, Ses H(1)w, (1), B (1), B (1), and select P.
2: for == 1,2, .... N do
3: run neural dynamics (in Algorithms 3 to 5 below according to P) until convergence.
4: W(k+1) W(k) + pw (k)))))ak)))))
5: BS (k: + Sy
(L)
Su By (k))))) BYy (k)) (k))
6: Adjust pw (k + 1) if necessary.
7: end for
Algorithm Online CorInfoMax neural dynamic iterations: sources in unit simplex
1: Initialize Vmax ct. Fl (1), (1), X(1)(= = 0 in general) and =
2: while (11y((;;)))))))))))) v) y (k: 1)ll/ llll/////;;/)))) [b; > <t) and v < Vmax do
3:
Vo(A) v)) = 7y By (k))));;)) 7o Be (k)e(k;)
4:
y(k; 1) ReLU (y(k: + m V (((p())) ((v)
5:
Vack ((y)k()) = [|u(k; 1)ll
6:
A(k; - 1) A(k; (n( ( ((r((k))) ((o(k(t)))
7:
== = 1, and adjust (1), nA (v) if necessary.
8:
end while
Algorithm Online CorInfoMax neural dynamic iterations: sparse sources
1: Initialize Vmax ét, Ty (1), (1), X(1) - 0 in general) and
2: while (11y((:: v) (k: 1)ll//]((k:>]]] > 4)) and v < Vmax do
3:
Te v)) "y, (b (b))(k))) (B (k)e((())
4:
V Cx(x)(yy(k;v)) = 1ly(k;++++1 w
y(k; + STM (y(k; ( (()))) -(v))
5:
6:
x(k; ReLU (A(k;)) Vxn v)))
7:
+ 1, and adjust m/ (4), (v) if necessary.
8:
end while
22

---

Published as a conference paper at ICLR 2023
Algorithm 5 Online CorInfoMax neural dynamic iterations: antisparse sources
1: Initialize Vmax, Et, Ty (1) and v/ =
2: while (1ly(k; v) yk;;v 1)1l//ll(( > Ex) and v < Mmax do
3: V Ty v)) = -y By (k)g((;)) - -c BK
4: y v + 1) 01 (v(k;)) + my(u) V ((x)T((((:
5:
== + 1, and adjust (v)) necessary.
6: end while
Algorithm 6 Online CorInfoMax neural dynamic iterations for Canonical Form
Initialize Vmx Cr fl (1), (1)),, X(1) = general) and v Ex) and v < Mmax do
2: while (1ly(k; v) yk; /lll///((k;)))))
3:
T(()) v)) = = (k) (R))) A
4:
y(k; + y(( (y()) ((t))
5:
(y(x)(y((:vv)) -App(())))==)))(()))) by
6:
A(k;v+ ReLU (A(I:V) [ll C(y(k:)))
7: V + 1, and adjust f/y (1), (v) if necessary.
())))
8:
end while
D.2 ADDITIONAL NUMERICAL EXPERIMENTS FOR SPECIAL SOURCE DOMAINS
D.2.1 SPARSE SOURCE SEPARATION
In this section, illustrate blind separation of sparse sources, i.e. s(i) Be, Vi. We consider
sources and m 10 mixtures. For each source, we generate 5 x 10 samples each realization of the experiments. We examine two different experimental factors: 1) output SINR performance function of mixture SNR levels, and 2) output SINR performance for different distribution selections for the entries of the mixing matrix.
For the first scenario with different mixture SNR levels: the sources are mixed through a random matrix A IR 10x5 whose entries are drawn from i..d. standard normal distribution, and the mix- tures are corrupted by WGN with 30dB SNR. We compare our approach with the WSM (Bozkurt al. 2022), L.DIIffMMxx (Erdogan 2022) and PMF (Tatli & Erdogan, 2021 algorithms and sualize the results in Figure 8 We also use the portion of the mixtures to train batch LD-InfoMax and PMF algorithms. Figure Sa illustrates the SINR performances these algorithms for different input noise levels. The SINR results of CorInfoMax. LD-InfoMax. and PMF are noticeably close to each other, which almost equal to the input SNR. Figure 8b illustrates the SINR convergence of the sparse CorInfoMax network for the 30dB mixture SNR level function of update iterations. Based on this figure, we can conclude that the proposed CorInfoMax network converges robustly and smoothly.
23

---

Published as a conference paper at ICLR 2023
Sparse Source Separation SINR Results
Online CorinfoMax Sparse SINR Convergence
30
30
25
SINR (6))
20
15-
15
CorinfaMax
10
WSM
10
5
LD-infoMax
PMF
f0.0
12.5
15.0
17.5
20.0
22.5
25.0
27.5
30.0
8.0
2.5
5.0
7.5
10.0 12.5
15.0 17.5
20.0
Input (dB)
Number Iterations 25000
(a)
(b)
Figure 8: SINR performances of CorInfoMax (ours), LD-InfoMax, PMF, and WSM (averaged over 50 realizations) for sparse sources: (a) output SINR results (vertical axis) with respect to input SNR levels (horizontal axis), (b) SINR (vertical axis) convergence plot as function of iterations (horizontal axis) of sparse CorInfoMax for the 30 dB SNR level (mean solid line and standard deviation envelopes).
the second experimental setting, we examine effect of the distribution choice for generating the random mixing matrix. Figure 9 illustrates the box plots of SINR results for CorInfoMax, LD-InfoMax, and PMF for different distribution selections to generate the mixing matrix, which are (0, 1), 1], 2, I.(0, and I. is the Laplace distribution. 1). where N is normal distribution, U is uniform distribution, observable that the performance CorInfoMax is robust against the different distribution selections of the unknown mixing matrix, while its performance is on par with the batch algorithms LD-InfoMax and PMF
Online CorlnfoMax Results
LD-InfoMax Results
PMF Results
34
32
32
32
31
SIRR (dB)
30
30
30
29
29
29
27
27
27
26
26
25
11
21
((0.1)
AT0, 1)
11-1111 11
112,22 21
00 1)
A90, 1)
M-1,11 11
1f-2.21 21
((0.1)
.10.1
Distribution Entries Mixing Matrix
Figure 9: SINR performances of CorInfoMax (ours), LD-InfoMax, and PMF with different distribu- tion selections for the mixing matrix entries and for sparse sources. The horizontal axis represents different distribution choices, and the vertical axis represents the SINR levels.
D.2.2 NONNEGATIVE SPARSE SOURCE SEPARATION
We replicate the first experimental setup in Appendix D.2.1 for the nonnegative sparse source separation: evaluate the SINR performance of the nonnegative sparse CorInfoMax network for different levels of mixture SNR, compared to the batch LD-InfoMax algorithm and the biologically plausible WSM neural network. these experiments, n 5 uniform sources in B are randomly mixed generate
10 mixtures. Figure 10a illustrates the averaged output SINR performances of each algorithm with a standard deviation envelope for different input noise levels. Figure 10b] we observe the SINR convergence behavior of nonnegative sparse CorInfoMax function of update iterations. Note that CorInfoMax outperforms the biologically plausible neural network WSM for all input SNR levels, and its convergence is noticeably stable.
24

---

Published as a conference paper at ICLR 2023
Nonnegative Sparse Source Separation SINR Results
Online CorinfoMax Nonnegative Sparse SINR Convergence
30
30
25
25
SINR (8))
20
15
15
10
CorinfoMax
10
5-
WSM
InfoMax
f0.0
12.5
15.0
17.5
20.0
22.5
25.0
27.5
30.0
8.0
2.5
5.0
7.5
10.0 12.5
15.0
17.5
20.0
Input (dB)
Number Iterations 25000
(a)
(b)
Figure 10: SINR performances of CorlnfoMax (ours), LD-InfoMax, and WSM for nonnegative sparse sources: (a) the output SINR (vertical axis) results with respect to input SNR levels (horizontal axis), (b) the SINR (vertical axis) convergence plot function of iterations (horizontal
axis) nonnegative sparse CorInfoMax for the 30dB SNR level (mean solid line and standard deviation envelopes).
D.2.3 SIMPLEX SOURCE SEPARATION
We repeat both experimental settings in Appendix D using the CorInfoMax network in Figure Figure 11a shows the output SINR results of both the online CorInfoMax and the batch LD-InfoMax approaches for different mixture SNR levels. Even though simplex CorInfoMax is not as successful as the other examples (c.g., sparse CorlnfoMax), in terms the closeness of its performance to the batch algorithms, still has satisfactory source separation capability. Similarly to the sparse network examples, its SINR convergence is fast and smooth, as illustrated in Figure 1lb
the blind separation of simplex sources
Simplex Source Separation SINR Results
Online CorinfoMax Simplex SINR Convergence
30
25
25
20
20
1.5
10
10
5
CortnfoMax
Infonfaaxx
0
12.5 Input SNR (dB)
15.0
17.5
20.0
22.5
25.0
27.5
30.0
8.0
17.5
Number of Iterations 25000
(b)
Figure 11: SINR performances of CorInfoMax (ours) and LD-InfoMax (averaged over 50 realiza- tions) for unit simplex sources: (a) the output SINR (vertical axis) results with respect to the input SNR levels (horizontal axis), (b) SINR (vertical axis) convergence plot function of iterations (horizontal axis) of simplex- CorInfoMax for 30dB SNR level (mean solid line and standard devia- tion envelopes).
Figure 12 shows the box plots of the SINR performances for both CorInfoMax and LD-InfoMax approaches with respect to different distribution selections for generating the random mixing matrix. Based on this figure, we can conclude that the simplex CorInfoMax network significantly maintains its performance for different distributions for the entries of the mixing matrix.
25

---

Published as a conference paper at ICLR 2023
Online CorInfoMax Results
LD-InfoMax Results
34
34
SINR (dB)
26
26
24
24
22
22
N10, 1)
M-1, 11
A1o. 1)
1f-1, 11
1f-2, 21
c(0.1)
M-2,
c(0,
Distribution the Entries of Mixing Matr
Figure 12: SINR performances of CorInfoMax (ours) and LD-InfoMax with different distribution selections for the mixing matrix entries (averaged over 50 realizations) and unit simplex sources. The horizontal axis represents different distribution choices, and the vertical axis represents the SINR levels.
D.2.4 SOURCE SEPARATION FOR POLYTOPES WITH MIXED LATENT ATTRIBUTES
In this section, we demonstrate the source separation capability of CorInfoMax on an identifiable polytope with mixed features, which is a special case of feature-based polytopes in 2 We focus the polytope
81, 82,
84
-1, 1), 83, 85
E
[0, 1]
Pex
$ R
81
82
(A.19)
52
1,
53
1
85
whose identifiability property is verified by the identifiable polytope characterization algorithm pre- sented in [Bokurrt & Erdogan (2022). We experiment with both approaches discussed Section 3.3 and Appendix C.5 For the Seature-based polytope setting introduced in Appendix C.5) the out- put dynamics corresponding to Pen is also summarized as an example. This polytope can also be represented as the intersection of 10 half-spaces, that is, Pez {s E RT Aps & bp} where Ap R 10x5 and bp E IR Therefore, using Ap and bp, we can also employ the neural network illustrated Figure Ta with 10 inhibitory neurons.
For this BSS setting, we generated the sources uniformly within this 5 dimensional polytope P where the sample size is 5 x 10 The source vectors are mixed through random matrix E IR 20x3 with standard normal entries. Figure 13a and 13b show the SINR convergence of CorInfoMax networks based on the feature-based polytope representation in 2 and the H-representation ( respectively, for the mixture SNR level of 30dB. Moreover, Figure 13c and convergence curves for the SNR level of 40dB. In Table 2 we compare both approaches with the batch algorithms LD- and PMF
their SINR
26

---

Published as a conference paper at ICLR 2023
Online CorinfoMax Special
Onlime CortnicMer Special Polytope SIMR Convergence
SINR Convergence
25
(68)
20
20
SN4
SIA
5
5-
6.0
5.0
10.0
15.0
20.0
0.0
5,0
10.0
15.0
20.0
Number of Iterations 25000
Number Iteeation/ 25000
(a)
(b)
Canonical Polytope
Online
Special Palytope
35
25
25
10
10-
8.0
9.0
Iterations 25000
Iterations 25000
(c)
(d)
Figure 13: SINR performances of CorInfoMax networks on Pex with mean-solid line with standard deviation envelope (averaged over 50 realization) for the polytope with mixed latent attributes: (a) SINR convergence curve for CorInfoMax feature-based polytope formulation with 30dB mixture for CorInfoMax canonical formulation with 30dB mixture SNR, (c) SINR convergence curve for CorInfoMax feature-based polytope formulation with 40dB mixture SNR, (d) SINR convergence curve for CorInfoMax canonical formulation with 40dB SNR.
SNR, (b) SINR convergence
Table 2: Source separation averaged SINR results on Pex for the CorInfoMax (ours), CorInfoMax
Canonical (ours) LD-InfoMxx, and PMF algorithms (averaged for 50 realizations).
CorInfoMax Canonical
Algorithm
CorInfoMax
LD-InfoMax
PMF
SINR ( (\w 30dB SNR)
26.55
24.85
30.28
27.68
SINR () ((w 40dB SNR)
30.93
26.19
38.50
31.48
D.3 APPLICATIONS
We present several potential applications of the proposed approach, for which we illustrate the usage of antisparse, nonnegative antisparse and sparse CorInfoMax neural networks. This section demon- strates sparse dictionary learning, source separation for digital communication signals with 4-PAM modulation scheme, and video separation.
27

---

Published as a conference paper at ICLR 2023
Online CorinfoMax 4-PAM SINR Convergence
25
SINR (dB)
20
15
10
8.0
1.5 Number of Iterations 25000
(a)
(b)
Figure 14: (a) Sparse dictionary learned by sparse CorInfoMax from natural image patches, (b) The SINR (vertical axis) convergence curve for the 4-PAM digital communication example function of iterations (horizontal axis) which is averaged over 100 realizations: mean-solid line with standard deviation envelope.
D.3.1 SPARSE DICTIONARY LEARNING
We consider sparse dictionary learning for natural images, model receptive fields in the early stages of visual processing (Olshausen & Field 1997). In this experiment, we used 12 x 12 pre- whitened image patches input to the sparse CorInfoMax network. The image patches from the website httpst//htt//www.rttt...rr wwwpetttrrrgbuuu/sppseeeeee
obtained The inputs are vectorized to the shape 144 Figure [4a illustrates the dictionary
before feeding to the neural network illustrated in Figure 2 learned by the sparse CorInfoMax network.
D.3.2 DIGITAL COMMUNICATION EXAMPLE: 4-PAM MODULATION SCHEME
One successful application of antisparse source modeling to solve BSS problems is digital com- munication systems Cruces 2010); Erdogan (2013). In this section, verify that the antisparse CorInfoMax network in Figure 4 can separate 4-puss-ampiruue-moduhhi
(4-PAM) signals with domain (-3, 1,,1,3, (with uniform probability distribution). We consider that digital com- munication (4-PAM) sources with 10" samples transmitted and then mixed through a Gaussian channel to produce 10 mixtures. The mixtures can represent signals received at some base station antennas multipath propagation environment. Furthermore, the mixtures are corrupted by WGN that corresponds to SNR level of 30dB. We feed the mixtures to the antisparse CorInfoMax network illustrated input. For 100 different realization of the experimental setup. Figure 14b illustrates the SINR convergence as a function of update iterations. We note that the proposed approach dis- tinguishably converges fast and each realization of the experiments resulted in zero symbol error rate.
D.3.3 VIDEO SEPARATION
We provide more details on the video separation experiment discussed in Section 4.2 Three source videos used in this experiment from the website https://www.pettt www.wwwwl..cm...ccoc which are free to download and use. Videos are mixed linearly through a randomly selected nonnegative 5 x 3 matrix
1.198
1.020
1.2737
0.367
1.364
0.901
A
0.100
0.869
0.627
1.257
0.859
0.015
0.957
0.789
0.592
illustrate the last RGB frames of the original and mixture videos, respectively. To train the nonnegative antisparse CorInfoMax network in Figure for the separation of the videos, we followed the procedure below.
to generate 5 mixture videos. Figures 15a and 15b
28

---

Published as a conference paper at ICLR 2023
In each iteration of the algorithm, randomly choose pixel location and select pixels from the color channels of all mixture videos to form mixture vector of size 1,
We sample 20 mixture vectors from each frame and perform 20 algorithm iterations per frame using these samples.
The demonstration video contains three rows of frames: the first row contains the source frames, the second row contains three of the five mixture frames, and the last row contains the network outputs for these mixture frames, Demo video is located at (https:/figshare wwt.c//////aaaaccc wtt..o///utto222222223330533333 http:://iggaarc.com/////tt222222
It also included in supplementary files. If we use the separator matrix the CorInfoMax network, which is estimate for the left inverse of A, to predict the original videos after training, we obtain PSNR values 35.600 25,60dB, 48.07 07dB7,d and 44.58dB for the videos, which are calculated as the average PSNR levels of the frames. Figure 15c illustrates the final output frames of the CorInfoMax.
(b)
(C)
Figure 15: Video separation example: the final frames of (a) sources, (b) mixtures, and (c) the outputs of the nonnegative antisparse CorInfoMax network.
D.3.4 IMAGE SEPARATION
Related to the example the previous section, we consider an example on the blind separation of correlated images from their linear mixtures. In this experiment, we consider 3 RGB natural scenes with sizes 454 x 605 that are illustrated in Figure 16a The Pearson correlation coefficient for these sources P12 0.076, P13 0.262, P23 0.240, respectively. The image sources mixed through a random matrix A E 5 whose entries are drawn from i.idd. standard normal distribution. Moreover, the mixtures computted with WGN corresponding 40dB SNR. The
29

---

Published as a conference paper at ICLR 2023
mixture images are demonstrated in Figure 16b
and the mixing matrix for this particular example is
-0.363
0.650
1.757
1.100
1.568
1.487
A
1.266
0.032
-0.417
0.822
0.643
1.260
0.023
-0.752
0.661
We experiment with the proposed antisparse CorInfomax and biologically plausible NSM and WSM networks, and batch ICA-InfoMax and LD-InfoMax frameworks to separate the original sources, and Figures [66 16g show the corresponding outputs. We note that the residual interference ef- fects in the output images of ICA-InfoMar algorithm is remarkably perceivable, and the result- ing PSNR values are 18.56dB, 20.52dB, and 21.06dB, respectively. The NSM algorithm's 's outputs visually better whereas some interference effects are still noticable, and the resulting PSNR values are 25.30dB, 49dB 25dB The visual interference effects in the WSM algorithm'ss 's outputs are barely visible, and, therefore, they achieve higher PSNR values of 27. 97.99dB, 29.71dB, and 31 .9.23. The batch I.D-InfMMax algorithm achieves the best PSNR performances which are 33.60dB, 31.99dB, and 33.62dB. Finally, we note that our proposed CorInfoMax method outper- forms other biologically plausible neural networks and the batch ICA-InfoMax algorithm, while its performance is on par with the batch LD-InfoMax algorithm, and its output PSNR values are 32.45dB 29.72dB, and 32.37dB. We note that ICA and NSM algorithms assume independent and uncorrelated sources, respectively, so their performances are remarkably affected by the correlation level of the sources. Typically, the best performance is obtained by L.DIIffMMax due to its batch nature and dependent source separation capability. Finally, we notice that the biologically plausi- ble WSM network is able to separate correlated sources whereas CorInfoMax performs better both visually and in terms of the PSNR metric. The hyperparameters used in this experiment for Corln- foMax are included in Appendix D.4] and the codes to reproduce each output included in our supplementary material.
(a)
Figure 16: Image separation example: (a) Original RGB images, (b) mixture RGB images, (c) ICA outputs.
30

---

Published as a conference paper at ICLR 2023
(d)
(I)
Figure 16: (d) NSM outputs, (e) WSM outputs, (f) LD-InfoMax Outputs, (g) CorInfoMax (ours) Outputs.
D.4 HYPERPARAMETER SELECTIONS
Hyperparameter selection has a critical impact on the performance of the neural networks offered in this article. In this section, we provide the our hyperparammter selections for the experiments provided in the article. These parameters are selected through ablation studies and some trials, which are discussed Appendix D.5
Table 3 summarizes the hyperparameter selections for the special domain choices provided in Sec- tion 2.1 Based on this table, we can observe that the hyperparameter sets for different special source domains generally resemble each other. However, there are some noticable domain-speciie changes in some parameters, such as the starting learning rate for neural dynamics (7y (1)) and the learning rate for the Lagrangian variable (1)x (z)).
For the other experiments presented in the Appendices D.2.4 and D.3 Table summarizes the corresponding hyperparameter selections.
31

---

Published as a conference paper at ICLR 2023
Table 3: CorInfoMax network hyperparameter selections for special source domains in
2.1
Source Domain
Hyperparameters
W(1== = B (1) 51, (1) 50001
= Bi
Su = 1- : Le 1 10-2 1w = x 10-2
1/mxx 500,
7y(v) = 009//, 4. 10
(11) I,
(1) = BC (1) 2000I
P Bi
Cy = 1 10
"e = = 100010111 /3, PW = 3 x 10 2
Vmax 500;
f/y
(v) = max u, 10 10-3,, "); 6t- = 10-6
(1) = I, (1) - BCr =10001
Cu = 1 - 10-2
-1 10-2 MW =3 10-2
P=B. B1
mmax 500,
m/
max 1,,10-3,
(M = 1, = 10
W (1) = Bby = 5I, (1) 10001
Ey = 1 - 10-2 -==1 - - MW = 3 x 10-2
p = =B,+
Vimax = 500,
Ny (y( = max(0.1/2m 10-3),
M7 (v) = 1, G == 10
(1) = I, (1) = 51, (1) = 10001
Eu = 1 10-2,
Ce = 1-1111 10-2, pW = 3 x 10-2
p = A
Mmax 500,
By (v) max 10-3),
17 (v( = 0.05, = 10
Table 4: CorInfoMax network hyperparameter selections for the polytope with mixed latent at- tributes in ) and application examples in Appendix D.3
Experiment
Hyperparameters
W(1) = 1,
B (1) = 1, BBa 10001
Polytope in (A.19) with Mixed Attributes (Canonical)
1 Sy =1 - 10 Se = 1 - 10-2. H/ 5 x 10
Mmmxx = 500,
my (v) = mmax(0.25/1m 10 10-4)
(1))
0.1, 6: 10
W(1) = I,
B (1) = 51, BC (1) = 2500I
Polytope in (A.19) with Mixed Attnbutes (Feature-baseds
Sy = 1 10-2
Ce = 10-2 P/W x 10-2
Vmax 500,
By (v) max (0.1 10-10), )
2x, (v)
Mag (v) = 1,
10-6
((1) =1 B (1) I, B (1)
200001
Sparse Dietionary
Sy - 10 3/7, 1- 1 Mw 0.25 x 10-3
Learning
V/mxx = 550,,
(y( = mmx((155
x 0.01), 10 83. 4 10
(2(2) 33 10-
((1) II
(1)= 5I,
(1) = 10001
Digital
Sy = 1- 1--22 < 11 1--0 joW 3 10-2
Communications
500, "/ (v) mmmxxx99//// 10-3) 4, 10-6
Vmax
W(1) = I, (1) I, (1² = 65I
Video
Sy = 11 10 /5, S.== p' = 6 10-
Separation
Vmax
500, My (v) = mmxx
10 10011-11-1 33, 6 = 10-6
W(1) = I, B (1) II (1) = 100I
Image
Cy - 10 /1, 6 0.5, MW 5x 10-2
Separation
Vmax = 500, "/ (v) max (0.5v.
10-3) 4 10-6
D.5 ABLATION STUDIES ON HYPERRPARAETER SELECTIONS
In this section, we illustrate an ablation study on selection of two hyperparameters for the non- negative antisparse CorInfoMax network: we experiment with selection for the learning rate of the
32

---

Published as a conference paper at ICLR 2023
feedforward weights ja and the initialization of the inverse correlation matrix _ For these hyperparameters, we consider the following selections:
μw (5 10-3 100 10-2, x 10-2),
B E {10", x 10²,5 5 x 10 10*}.
We consider the experimental setup in Section 4.1 for both uncorrelated sources and correlated sources with correlation parameter p 0.6. For the ablation study for jw, used fixed B as indicated in Table and vice versa.
Figures 7a and 17b illustrate the mean SINR with standard deviation envelope results with respect to jow for uncorrelated and correlated source separation settings, respectively. Note that although selection jow 5 10 seems be better for uncorrelated sources, its performance degrades for correlated sources as for some of the realizations, the algorithm diverges. We conclude that the selection jaw 3 x 10-2 is near optimal in this setting and obtains good SINR results for both correlated and uncorrelated sources. For the initialization of B Figures 17c and 17d illustrate the SINR performances of CorInfoMax for uncorrelated and correlated source separation experiments, respectively. Note that the selections BC- = 2000 and B 5000/ are suitable considering both settings.
30
30-
28
(66)
26
26
56.3
16:2
362
56:3
3e12
2e22
(a)
(b)
26
24.
22
163
263
Se3
169
163
263
3e3
164
B, Gain
8, Gain
(c)
(d)
Figure 17: CorInfoMax ablation studies on the hyperparameter selections of μw and B mean SINR with standard deviation envelopes corresponding to 50 realizations. (a) ablation study of uw on uncorrelated nonnegative antisparse sources, (b) ablation study of pw on correlated (with = 0.6) nonnegative antisparse sources, (c) ablation study of B on uncorrelated nonnegative antisparse sources, (d) ablation study of B on correlated (with p = 0.6) nonnegative antisparse sources,
D.6 ABLATION STUDIES ON THE EFFECT OF NUMBER OF MIXTURES
The number of mixtures with respect to the number of sources might be crucial for blind source separation problem, and it can affect the overall performance of the proposed method. To explore the impact of the number of mixtures formed experiments with varying number mixtures and fixed number sources for nonnegative antisparse, i.e., P Bo and sparse, i.e., P B1, source separation settings. In these experi-
the SINR performance of our proposed method,
per-
33

---

Published as a conference paper at ICLR 2023
mental settings, consider 5 sources, and change the number of mixtures gradually from 5 to 10. Figure 18a and 18b illustrate the overall SINR performances with standard deviation envelope for nonnegative antisparse CorlnfoMax and sparse CorInfoMax networks with respect to the number of mixtures, which are averaged over 50 realizations, respectively. For each realization, we randomly generate mixing matrix with i.i.d. standard normal entries. We observe that the performance of CorInfoMax networks monotonically improves as the number of mixtures increases. This aligns with the theoretical expectations that the condition of the random mixing matrix improve with in- creasing number of mixtures Chen & Dongarra (2055) which positively impacts the algorithm's 's numerical performance.
Number Mixtures
Number Mixtures
(h)
Figure 18: CorInfoMax ablation studies the impact of number of mixtures with fixed number of sources (5 sources). (a) illustrates the SINR performances of nonnegative antisparse CorInfoMax network as function of number of mixtures, (b) illustrates the SINR performances of sparse Cor- InfoMax network as a function number mixtures.
D.7 COMPUTATIONAL COMPLEXITY OF THE PROPOSED APPROACH
The complexity of proposed approach can be characterized based on the analysis of output dynamics and weight updates. For simplicity, we consider the complexity of the antisparse CorInfo- Max network Figure Let Vmaz represent the maximum count for the neural dynamic iterations.
Neural Dynamics' Complexity: For one neural dynamic iteration, we observe the following number of operations
Error calculation e(k; v) y(k; W(k)æ(k) requires multiplications,
Biy (k))((;;)) requires n multiplications,
(k)e(k;;)) requires multiplications B (k) 1
As result, Umar neural dynamics iterations require Vman (m ) Vmaziiin multiplications per output computation.
Weight Updates' Complexity: For the weight updates, we note the following number of operations
W + =)= W(k) + uw (k) requires 2mn multiplication,
Ey B&y (R))(()))()) B()()) requires n² + n² +n + +1
BKy 1) (k)
2y
n(n +1) 7n +2
multiplications.
Therefore, weight updates for learning require 2mn + 7n2+n+2
multiplications per input sample.
Taking into account both components of the computations, complexity dominated by the neural dynamics iterations which require approximately O( operations. This is in the same order as the complexity reported in Bozkurt al, (2022) for biologically plausible WSM, NSM, and BSM networks.
34


Reviews:
[{"review_id": "aMRd0Wn_zz", "reviewer": "Reviewer_hwJs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I thank the authors for providing a thoughtful reply addressing most of my concerns. Together with the clarifications I am now convinced that this work is a good contribution. I have adjusted my score accordingly.\n\nThe paper proposes a biologically plausible source separation method with local updates and batched online learning.\n\nI can see that the paper is based on a recent publication by Erdogan (2022). Generally, this work cites a large number of previous works by that author. I cannot judge how novel the contribution in this paper is compared to those previous works. The framework and mathematical formulation are extremely complex and I was not able, within reasonable time investment, to digest and asses the full breadth of it. The applications seem somewhat niche. And the only real data application does, as far as I can tell, not include the interesting setting of correlated sources. By the way, the same setting (in more interesting nonlinear scenarios) is also investigated in this (https://proceedings.mlr.press/v139/trauble21a.html) work.", "summary_of_the_review": "The work is hard to digest and it is not clear how incremental it is compared to previous work.", "strength_and_weaknesses": "The paper is well written, with a thorough mathematical derivation. It also cites many relevant existing works.\n\nThe paper is extremely dense, possibly incremental on previous work, somewhat niche.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NOm1jr17eWY", "reviewer": "Reviewer_C2ij", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper develops an online algorithm with local update rules for blind source separation using information maximization in terms of second-order statistics and knowledge of the domain of the sources.\n\nLike previous biologically plausible algorithms, the online algorithm can be instantiated using only Hebbian and anti-Hebbian learning in two or three layer networks. \n\nIn particular, when the domain of the sources are known to be a polytope (L1-ball for sparse vectors), L$\\infty$-ball for dense vectors, possibly non-negative (yielding simplex for the former), the Lagrangian of the constrained maximization problem is shown to induce different activation functions. \n\nResults show that the method is competitive to other prior work on synthetic problems where the true domain of the sources is known. ", "summary_of_the_review": "The paper proposes an online algorithm for blind source separation using a log-det based information maximization principle under a second order statistic.  The method builds from previous work, and is logical and appears correct. Paper does thorough job in describing how this leads to biologically plausible cases. But the clarity of the presentation at times could be improved. Overall novelty is mostly incremental. Results are detailed but challenging real world cases are not covered. The knowledge of the domain of the sources is required, and it is unclear how a user would be able to blindly select hyper-parameters. ", "strength_and_weaknesses": "Strengths:\nThe work builds off of a solid foundation for BSS using second-order statistics and is logically presented.\nThe method compares well to related work in ideal cases. \nThe local updates seem reasonable based on the assumptions. \n\nWeaknesses: \nThe figures of networks are not clear from their captions.\nThe overloaded notation is quite cumbersome at times.  \nThe method is not compared to other approaches on the real data.\nAlthough dependent sources are discussed a real world example of dependent sources is not illustrated. \nThere is a relatively large number of hyper parameters. It is not clear if the hyper parameter selection was done in a reproducible manner. I.e. if the result cannot be evaluated by a metric or qualitatively can the user practically tune the hyper-parameters? If so with what criteria? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4n_pg0bMFb", "reviewer": "Reviewer_LCHG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an online algorithm for blind source separation that is represented by a multi-layer network with biologically plausible weight/activity dynamics.", "summary_of_the_review": "Good paper with a solid technical and experimental contributions. I recommend acceptance.", "strength_and_weaknesses": "### Strengths\n\nTheoretically grounded algorithm for blind source separation.\n\nDerivation of network dynamics that can implement this algorithm online.\n\nGood performance despite the online setting and correlation matrix approximations.\n\n### Weaknesses\n\nWriting is lacking in some parts (see below), but contribution-wise I don't see any issues. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster