Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: I thank the authors for providing a thoughtful reply addressing most of my concerns. Together with the clarifications I am now convinced that this work is a good contribution. I have adjusted my score accordingly.

The paper proposes a biologically plausible source separation method with local updates and batched online learning.

I can see that the paper is based on a recent publication by Erdogan (2022). Generally, this work cites a large number of previous works by that author. I cannot judge how novel the contribution in this paper is compared to those previous works. The framework and mathematical formulation are extremely complex and I was not able, within reasonable time investment, to digest and asses the full breadth of it. The applications seem somewhat niche. And the only real data application does, as far as I can tell, not include the interesting setting of correlated sources. By the way, the same setting (in more interesting nonlinear scenarios) is also investigated in this (https://proceedings.mlr.press/v139/trauble21a.html) work.
strength_and_weaknesses: The paper is well written, with a thorough mathematical derivation. It also cites many relevant existing works.

The paper is extremely dense, possibly incremental on previous work, somewhat niche.


clarity,_quality,_novelty_and_reproducibility: It is not clear how important the biological plausibility is and that does not seem be further explore by the paper.

Nits:

Both the title and the abstract are very hard to understand and one is left wondering what the paper is about. Latent causes are not defined, domains are not defined, what are presumed sets?

Second paragraph: The fact that biological receptive fields look like ICA filters should not be confused with the much stronger claim that the brain is doing BSS.

All figures need more explanation in the caption.

1.1.1 second to last sentence: ‘while’ ? ‘they’; autocorrelation

2.1.iii superfluous dot?

2.1 after list the “” quotation marks are wrongly formatted

Fig3 define the axis labels in the caption
summary_of_the_review: The work is hard to digest and it is not clear how incremental it is compared to previous work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an online algorithm for blind source separation that is represented by a multi-layer network with biologically plausible weight/activity dynamics.
strength_and_weaknesses: ### Strengths

Theoretically grounded algorithm for blind source separation.

Derivation of network dynamics that can implement this algorithm online.

Good performance despite the online setting and correlation matrix approximations.

### Weaknesses

Writing is lacking in some parts (see below), but contribution-wise I don't see any issues. 
clarity,_quality,_novelty_and_reproducibility: ### Clarity

Overall, the paper is decently written. However, some of the results/derivations in my opinion should be moved to the appendix to leave more space. 

This space can be used to, for instance, write down dynamics of all weights and activations for the whole network in one place. Right now the dynamics are presented one by one, which is somewhat hard to follow.

Minor comments:

\hat R in Eq.3 should be defined in the main text.

(personal preference) Adding Eq. when referencing equations is more readable.

Labelling your methods as (ours) in tables/plots should improve readability.

### Quality 

The paper is technically solid. Moreover, the experiments are very extensive and cover several source separation setups, showing decent performance of the proposed algorithms.

### Novelty

The work is, to my knowledge, novel.

The discussion around Eqs. 14-15 reminds me of the FORCE rule [Sussillo and Abbott, 2009], which ended up with the same issue of approximating recursive least squares (RLS) to make it more plausible. The RLS derivation is similar to what you have in Appendix B. Probably it’s worth mentioning as a background citation? I think they ended up with a similar more plausible approximation to RLS, so some approximations in that or the follow-up work could improve your algorithm too, although it might be out of the scope of this work. See, for instance,

Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning
GM Hoerzer, R Legenstein, W Maass, 2014

### Reproducibility
The code is provided but not checked. Additionally, the hyperparameters are listed in the appendix.

summary_of_the_review: Good paper with a solid technical and experimental contributions. I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper develops an online algorithm with local update rules for blind source separation using information maximization in terms of second-order statistics and knowledge of the domain of the sources.

Like previous biologically plausible algorithms, the online algorithm can be instantiated using only Hebbian and anti-Hebbian learning in two or three layer networks. 

In particular, when the domain of the sources are known to be a polytope (L1-ball for sparse vectors), L$\infty$-ball for dense vectors, possibly non-negative (yielding simplex for the former), the Lagrangian of the constrained maximization problem is shown to induce different activation functions. 

Results show that the method is competitive to other prior work on synthetic problems where the true domain of the sources is known. 
strength_and_weaknesses: Strengths:
The work builds off of a solid foundation for BSS using second-order statistics and is logically presented.
The method compares well to related work in ideal cases. 
The local updates seem reasonable based on the assumptions. 

Weaknesses: 
The figures of networks are not clear from their captions.
The overloaded notation is quite cumbersome at times.  
The method is not compared to other approaches on the real data.
Although dependent sources are discussed a real world example of dependent sources is not illustrated. 
There is a relatively large number of hyper parameters. It is not clear if the hyper parameter selection was done in a reproducible manner. I.e. if the result cannot be evaluated by a metric or qualitatively can the user practically tune the hyper-parameters? If so with what criteria? 
clarity,_quality,_novelty_and_reproducibility: 
Clarity: 
In the abstract, the understanding of domain and "their presumed sets" needs to be made more explicit. 

Figure 1 is very detailed but it is presented without sufficient introduction to the notations, these should be in the caption. 

Ambiguous 'they' in "or example, the lateral connections of the outputs in Bozkurt et al. (2022) are based on the output autocorrelation matrix, while they are based on the inverse of the output autocorrelatin matrix" (also note the typo).

Equation 3e can be interpreted as a surrogate for H(Y) - H(Y|X), which is discussed in the appendix but should be included in the body. 

Section 4.2 
"to exclude the negative frame effect"

Novelty:
The work builds on an existing objective and follows other biologically plausible networks. Thus, this is incremental novelty. 

Quality:
The derivations appear sound and the overall presentation is logical. 

Although dependent sources are discussed a real world example of dependent sources is not illustrated. 

The case of separation with as many observations as sources is not discussed.   I.e. 5 mixtures of 3 sources is demonstrated. How well does it work for 3 mixtures of 3 sources? 

Reproducibility: 
There is a relatively large number of hyper parameters. It is not clear if the hyper parameter selection was done in a reproducible manner. I.e. if the result cannot be evaluated by a metric or qualitatively can the user practically tune the hyper-parameters? If so with what criteria? 
summary_of_the_review: The paper proposes an online algorithm for blind source separation using a log-det based information maximization principle under a second order statistic.  The method builds from previous work, and is logical and appears correct. Paper does thorough job in describing how this leads to biologically plausible cases. But the clarity of the presentation at times could be improved. Overall novelty is mostly incremental. Results are detailed but challenging real world cases are not covered. The knowledge of the domain of the sources is required, and it is unclear how a user would be able to blindly select hyper-parameters.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper propose an information-theoretic framework for creating neural networks that separate latent sources in a dataset, including for correlated sources. The authors use the maximum correlative information transfer from inputs to outputs as the separation objective, with additional constraints. This leads to local learning rules for the networks that can learn good separation of sources.

The reviewers agreed that the paper is technically sound and provides a novel contribution.

There were some consistent concerns regarding the density of the paper and lack of clarity in parts, as well as a few more reviewer specific concerns about whether the results are incremental or whether the connections to biology are all that strong. However, after rebuttal, the final scores moved up a bit to provide a final average score of 6.33. Based on these scores, as well as the general consensus that the work is technically sound and provides at least some novel contributions, a decision for accept was reached.
justification_for_why_not_higher_score: Though the reviewers were generally positive, and hence I think it should be accepted, the paper is very dense and is arguably not incredibly novel. Thus, I don't think it is appropriate for a spotlight.
justification_for_why_not_lower_score: The work is technically sound and does make a clear contribution, so rejection would be inappropriate.
Decision: Accept: poster