Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present a biologically-plausible algorithm for blind source separation (BSS) of correlated input sources. By applying weight similarity matching (WSM) approach to the Det-Max optimization algorithm used for BSS, the authors show that they can derive 2-layered Hebbian neural networks that are able to separate correlated sources from linear mixtures. They then compare their methods against Independent-Components Analysis (ICA) and Nonnegative Similarity Matching (NSM) methods in two tasks: one with artificial signals and another with natural images. In the task with artificial signals, the authors show how their algorithms is robust against correlation in the input sources whereas ICA and NSM degrade as measured by SINRs. The authors report results for the task of separating mixtures of natural images that are in-line with the former results, and they cherry-pick an example that illustrated the superior performance of their algorithm.      


strengths_and_weaknesses: The paper offers a general framework for deriving neural-network from the Det-Max approach, which is a novel contribution. They are the first (to the best of my knowledge) to propose a biologically-plausible algorithm that separates correlated sources. Overall, seems a worth contribution in terms of novelty. In terms of quality, I think the paper is very solid, and there are no obvious errors that I can see. The only experiment that I am missing is one that compares WSM with algorithms that are able to separate correlated sources and reporting how the performance compares there. I think this could be a necessary addition to judge rating of the paper. The paper is fairly clearly written, but I would suggest to add a section that states more clearly what the contributions of the paper are. I like that the authors highlight the limitations of their approach. In terms of significance, I think the paper is relevant for understanding how brains may accomplish BSS and to investigate novel biologically-inspired algorithmic improvements that can help advance state-of-the-art methods of BSS with correlated sources.  
questions: Please add more experiments and report SNIR comparing against algorithms that are able to separate correlated sources. 
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work focuses on the BSS problem and solves it by imposing some geometrical priors on the sources via an online weighted similarity matching algorithm (WSM). 
Since WSM does not use statistical independence of the sources to recover them, sources can be recovered even if they are correlated.
WSM is benchmarked on 2 datasets (a synthetic and a toy dataset) and is shown to yield better performance than ICA or non-negative similarity matching (NSM).
strengths_and_weaknesses: Strength:
- A general framework that applies to a large set of priors on the sources
- An online algorithm with local update rules that is therefore more biologically plausible

Weaknesses:
- Many hyper-parameters to set and no clear rules on how to set them (but the paper is transparent about this which is a good point)
- Experiments are on synthetic data and a toy dataset that does not really correspond to any realistic problem
- Missing comparison with Det-Max criterion

I quickly reviewed the code: it lacks documentation and unit tests are missing but is overall well structured and readable. I would still advice the authors to document every public function, make unit tests, examples and set up a continuous integration so that other researchers can easily build upon their work.
questions: - Hyper-parameters setting: 
There are many hyper-parameters to set in this method.  Did you perform a sensitivity analysis to see in which range they work ? Is there a way to find default values that would work for all kind of problems ? Can you explain how you chose the values in section E.2 and E.3 ?

- Experiments on synthetic data:
While the experiments on synthetic data show what they are meant to show, it would have been nice to see some experiments with real systems. ICA is used in many different settings: astronomy, neuroscience, finance (see the section 7 of Hyvärinen, Aapo, and Erkki Oja. "Independent component analysis: algorithms and applications." Neural networks 13.4-5 (2000): 411-430.). 
Did you try any experiments with real data ?

- Missing comparison with Det-Max criterion
According to your theorem 1: Problem (3) with Det-Max criterion and Problem (4) that yield the equations for WSM updates solve essentially the same problem. Therefore it would have seem natural to check whether they yield the same results in the synthetic experiments. Did you perform this comparison ? 
The Det-Max criterion is mathematically much simpler and seem easier to optimize. Why should practitioners use WSM instead ?
limitations: Ethical limitations are properly discussed
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work follows a recent line of work on formulating blind source separation problems as solutions of similarity matching objective functions. This work greatly expands existing work in the domain by proposing geometric interpretation and an objective function related to the Det-Max approach. Also, the formalism allows for the derivation of a biologically-plausible and online learning algorithm. Indeed, the model can be implemented by a two-layer neural network with local learning rules. 
strengths_and_weaknesses: The authors have covered a broad class of blind source separation problems. These problems generalize the well-known problems for which there existed or not biologically plausible learning algorithms. There is a broad modeling literature on BSS, and a growing one on similarity matching, and these generalizations and extensions of both is very novel. Although it is very technical the manuscript is well written and is easy fairly easy to follow. 

I found three minor weaknesses in this paper that can be easily addressed. 

One is that the paper claims that the resulting algorithm is only, but only presents these results in the appendix. I would have appreciated that some space of the main paper be allocated to that as it is rather central to the paper. 

The second one is related to the context of the problem. The paper is at the interface of signal processing, machine learning, and neuroscience, and it is a bit much to ask the reader to be well versed in all the different BSS problems covered in the paper. A bit more context for each of the problems would help understand the importance of each of these problems and why building such biologically plausible would be useful. Are natural data mixed or present in the form presented in this paper? 

Finally, the work mainly compares to nonnegative similarity matching and infomax, but it could be interesting to see how the model compares to existing algorithms designed to solve the problem at hand, not only biologically inspired ones. 
questions: My questions relate to the minor weaknesses found above: 

Can you propose a concise presentation of the online result showing that your model can operate in that setting? 

Can you explain why would biological plausible model has to solve such problems? 

Can you compare your model to existing algorithms that were designed to solve it, and not only bio-inspired ones? 
limitations: The authors have addressed the limitations of the paper in the last section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a novel neural network architecture to perform a blind source separation task, specifically addressing the case of correlated sources. The proposed architecture is shallow (2 or 3 layers) and derived in an online manner to maximize biologically plausibility. The proposed architecture is shown to solve a determinant-maximization problem, as proved in Theorem 1 (line 172), and to adequately solve two synthetically correlated source separation toy examples.
strengths_and_weaknesses: *Disclaimer: Due to my lack of expertise on the topic, I am not able to assess the soundness of the mathematical model nor the correctness of the theorem proof. I have also not read carefully the very long appendix (31 pages, 17 figures). Due to the length of the full manuscript, a journal might be a better publishing venue to benefit from more extensive reviews.*

### Originality
The blind source separation problem tackled by the paper is a well-known problem, addressed by a wide literature of methods. The originality seems to resides in the special case of correlated sources (as opposed to e.g. ICA which specifically assumes independent sources).

The proposed approach builds upon a recent framework called weighted similarity matching (WSM) and introduced in [15]. The difference with related works is very briefly discussed lines 70-74.

### Clarity
The paper is clearly written, but the relation to prior work is somewhat limited. A number of existing methods are listed, but the difference with the proposed method is not clearly explained. The paper would benefit from being more pedagogical about its different original contributions. 

### Quality
The numerical experiments to demonstrate the effectiveness of the model are quite limited. It would have been interesting to compare the proposed method with more methods.
questions: 1. What makes the proposed method well suited from correlated sources?

1. Figure 3/4 only compare the proposed method with ICA and NSM, despite a larger number of related works listed in section 1.1: NMF, SSMF, SCA, BCA, PMF, BSM. Why not comparing with these other methods?

limitations: - The paper stresses the computational cost of the proposed method, but does not give concrete examples. What is the computational cost of the proposed method (including hyperparameter tuning) in the two proposed numerical experiments? What is the computational cost of ICA and NSM?
review_title: null

Meta-review: metareview: This paper presents a method for blind separation of correlated sources, which is a challenging task. Applying the weight similarity matching approach to the Det-Max optimization, the authors develop a biologically-plausible two-layered neural network that can separate correlated sources from their linear mixture. All of reviewers agree that the paper is well written and has a solid contribution in BSS. The approach in this paper is a general framework that applies to various source domains. A downside is in experiments on real-world data, which has been improved during the author rebuttal period. Therefore, I am pleased to suggest the paper to be accepted.

confidence: Certain
Decision: Accept