Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a compartmental single-layer neural network model to be used in an online ICA algorithm utilizing a local synaptic update rule. The novel algorithm is based on the FOBI procedure for PCA by restating the objective function using similarity matching. As opposed to hebbian update rules, the synaptic updates used in the present paper incorporate a time-varying modulating factor which is computed as a function of the total output activity. Biological plausibility of this relaxed locallity constraint is justified by hinting at experimential results showing the importance of a third factor in the outcome of hebbian plasticity. The approach is verified using three different experiments including synthetic and real-world data.
main_review: Originality: 
The search for a local learning rule implementing ICA has a long history and many approaches have been investigated in the past. The presented approach is comparable with the work of Isomura and Toyoizumi which is also noted in the paper. While the mathematical details elude the reviewer, the implications and resulting learning rules seem quite similar. The main difference appears to be trading off a constant "error baseline" for lateral inhibitory connections. A more thourough comparison of the two approaches could further improve the quality of the paper.

Quality: 
The submission seems to be technically sound. Claims are supported by mathematical proofs and adequate experimental results. The authors are honestly stating the weaknesses of their approach.

Clarity: 
The approach is laid out in a clear and understandable fashion. Included figures are helpful for understanding the resulting NN structure and the correspondence to the FOBI procedure.

Significance:
Introducing yet another biologically plausible method for ICA might be considered of mediocre importance. Nonetheless, the work may advance our understanding of the inner workings of biological neural networks by linking a novel objective function to a simple network motif.

limitations_and_societal_impact: Limitations and possible remedies are listed. Due to the specific mathematical nature of the the paper its societal impact is unlikely to exceed the impact of the field as a whole.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This is a well-written paper containing novel ideas. The paper is motivated by an interesting question: how might fundamental computations such as independent component analysis (ICA) be performed by brains? The authors develop a neat algorithm to perform ICA. The paper claims that the developed algorithm is biologically plausible, i.e. might run in a similar form in a biological brain. 

# Update (2021-08-20)

Following the author's response from 2021-08-20, all my questions are answered. I had misunderstood the semantics of $x_t$. I updated my score. 

main_review: # Update (2021-08-20)

Following the author's response from 2021-08-20, all my questions are answered. I had misunderstood the semantics of $x_t$. I updated my score. 


# Overview

This is a well-written paper containing novel ideas. The paper is motivated by an interesting question: how might fundamental computations such as independent component analysis (ICA) be performed by brains? The authors develop a neat algorithm to perform ICA. The paper claims that the developed algorithm is biologically plausible, i.e. might run in a similar form in a biological brain. 

My main criticism of the paper is that unless there is a misunderstanding on my part, the described algorithm may not be biologically plausible. I would be grateful if the authors could respond to my questions Q1, Q2 and Q3 below. If the authors argue successfully in this review process and in the paper that the proposed algorithm is biologically plausible, I will change the review score that I am assigning today. 

# Questions for the authors

- Q1. Unless I misunderstood the paper, the algorithm that the authors propose and claim to be biologically plausible does not appear biologically plausible to me. In Algorithm 1 and Section 5.1, the authors explain that computing independent components involves updating the “weight matrix W encoded by the feedforward synapses connecting the input neurons to the output neurons”. This would imply that synapses re-learn completely new weights every time that the set of neurons computes independence components. It would be very surprising if the brain worked in such a manner. To be concrete: while listening to a conference speaker in a full conference room, my auditory system constantly has to separate the speaker’s voice from the background noise. As this computation happens, my auditory system’s synapses do not need to re-wire themselves in every fraction of a second, as I process a new chunk of audio signal. Instead, my auditory system in the brain developed some kind of noise filter when I was an infant, effectively learning synaptic weights that let me separate human voice from background noise across a wide range of environments. These weights may have changed somewhat over the years, but they certainly do not need to update every moment that I am processing auditory information. In essence, the authors are using (synaptic) weights that we know to learn slowly as fast-updating working memory. A biological neuron cannot learn fast enough to support running Algorithm 1 in real-time. 

- Q2. It would be useful if the authors could add details on how the proposed algorithm would map to known structures in say the human brain. The authors mention that “Various brain areas, such as the early visual, auditory, and olfactory systems, are known to identify 17 sources from their mixtures effortlessly [20, 85, 5, 52], an unsupervised task also known as blind 18 source separation (BSS) [19].” Since NeurIPS is mainly a machine learning conference, it would be help if the authors could expand on this sentence, survey in more detail what relevant information is known, and describe where they see the proposed algorithm fit in. For say visual processing, would the proposed algorithm run in the neurons of the primary visual cortex (V1)? 

- Q3. Have the authors benchmarked against any other ICA implementations? 


# Other suggestions and ideas for improvement


- Algorithm 1 contains notation that I find odd: just after “run the following until convergence:”, it looks like we assign to a derivative. Is that a typo? The line should probably be something closer to equation (16). 


- The proposed algorithm is very specific. Is there biological evidence suggesting that the brain is running an algorithm like the one proposed? 

- Reference [5] cited by the authors suggests that auditory scene analysis might involved not only the bottom-up kind of computing that is present in the proposed algorithm, but also top-down processing: ”In humans, auditory scene analysis involves both bottom-up and top-down processes (reviewed in Bregman, 1990; Carlyon, 2004; Feng & Ratnam, 2000; Näätänen, Tervaniemi, Sussman, Paavilainen, & Winkler, 2001). Bottom-up mechanisms are “stimulus-driven,” meaning that they operate only or primarily on cues present in the acoustic signal itself; they are largely automatic and obligatory, meaning that they do not critically depend on attention (although this is currently a disputed issue). In contrast, top-down processes depend on a listener's prior experience and expectations, and thus involve higher-level cognitive processes, such as learning, memory, and attention.” It would be useful to point out that a bottom-up algorithm like the one described in this paper would likely only constitute one part of a more complex system to perform e.g. signal processing computations. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a biologically plausible neural network (BPNN) with an independent component analysis (ICA) objective loss function to solve blind source separation problem. The main contribution of this paper lies in that, it extends BPNN for principal component analysis (PCA) [56] to BPNN for ICA with the fourth-order blind identification (FOBI). Simulation experiments demonstrate that the proposed approach is effective in solving blind source separation problem. 
main_review: The propose approach is interesting and has innovations both theoretically and algorithmically. The paper is well organized and presented. 

One shortage of the paper lies in that the experiment part is not strong. The experiments contain three datasets of a synthetic data, a speech data, and an image data. The results are mostly presented with illustrations rather than quantitative criterions. It also lacks of comparisons with other approaches. Adding quantitative evaluations and comparisons can help demonstrate the advantages of the proposed approach.

It is also interesting to ask what are the possible applications of the proposed approaches? What can we benefit from the biologically plausible neural network in the blind source separation problem? Can the proposed model help explain biological mechanisms in the brain?

============================

Post-rebuttal:

The authors have addressed most of my comments. I have raised my score from 6 to 7.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a normative study by combining the inspirations from fourth-order blind identification and similarity matching to derive a novel learning rule to achieve independent component analysis in a single layer recurrent network. The derived learning rule only depends on the overall neuronal responses in the network, rather than the detailed population activity pattern, which makes the learning rule more biologically plausible compared with previous work. The paper also justifies some possible neural substrates to access the information of overall neuronal activities in a neuronal population. 
main_review: This work is based on previous work of similarity matching and marries it to fourth-order blind identification to achieve independent component analysis. The derived learning rule is novel, and the mathematical analysis is solid. The whole paper is structure-wise and well written. Overall I think it is a good normative study worth publishing.

Major concerns
* Motivation of minimax optimization.
When I read the paper I feel the flow has a jump at section 4.1 in that I cannot get the motivation of converting the original minimized objective function into a minimax objective function, and the motivation of proposing the particular form of minimized objective. Is this minimax objective a necessary step to implement the algorithm in a network? Or the previous similarity matching algorithm optimize a minimax objective function? I wish the authors could add more motivations in a revised manuscript.


Minor concerns
* Eq. 5: is $\Lambda$ a parameter that needs to be optimized? Reading through the whole paper I don’t see how $\Lambda$ is constrained.

* Eq. 13: I can tell $\alpha_t  = |y_t|^2$ resulted from a rotation matrix defined in step 2b. However, I don’t see why $\alpha_t = |s_t|^2$. Is there an implicit assumption that the mixing matrix A (Eq. 1) is also a rotation matrix?

* line 155: Eq. 10 --> Eq. 8

* Eq. 18: the authors called this a thresholded nonlinearly modulated Hebbian rule. I don’t see how the thresholding process comes from. I know the $\Lambda$ determines LTP and LTD but there seems to be no thresholding like all or none process similar with thresholding in spike generation. 

* The Eq. 6 and definition of $\Gamma_x$ at line 140 can be simplified as the result appearing Eq. 12, which could simplifies the math equations a lot.

* Comparing Fig. 1B and neural dynamics shown in Eq. 16, the recurrent connection weight should be $I-M$ instead of $M$ since typically there is a leak term.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work proposes a novel similarity-preserving objective function for targeting ICA. They utilize this objective function to derive a gradient descent-based learning algorithm that takes advantage of this objective function. This algorithm is mapped to a single-layer neural network that is able to separate independent sources without pre-processing. They compare the proposed learning algorithms with oja-based methods, BCM, and Modulated Hebbian learning.
main_review: The strengths of this work are summarized as follows:
- The work proposes a novel perspective on a topic that has not been heavily explored.
- The development of biologically plausible neural networks could provide valuable insight into the brain’s computational properties. In addition, a biologically plausible method of ICA is particularly valuable.
- The theoretically grounded algorithm successfully integrates insights from the FOBI procedures into a neural network structure. The experimental results indicate that the implemented neural network can successfully perform ICA.

However, the observed weaknesses of the paper are summarized as:
- The experimental results presented demonstrates that the methodology can recover the original sources from a Mixed signal. But it is difficult to quantify the degree of that success for comparison with other methodologies. The authors should include a quantitative evaluation of the algorithm's effectiveness and quality of the recovered signals.
- While the work contrasts the proposed work with the Oja-based, BCM and Modulated Hebbian methodologies, the work does not present experimental comparisons to highlight the benefits of the proposed approach. The authors should highlight these distinctions.
- The authors utilize the proposed scheme only in a single layer architecture. The authors should demonstrate the scalability of the proposed scheme using a deeper model. 

In addition to the above comments, we hope the following can help the authors further improve their work.
- The font size of some text in both figures should be increased.
- While computational restrictions are not necessarily critical in the target scenario, it would be good to provide such information to evaluate the algorithm’s overhead properly.
- Since the two-compartment neural model is uncommon, it would be good to include a concrete mathematical model of its functionality.
- The authors make the claim that a minimal biological neuron model must adhere to the online learning and local update rule constraints. It would be good for the authors to motivate this assumption as it is not self-evident.

limitations_and_societal_impact: The weaknesses and limitations of the work are discussed in the main review. The work would have any obvious direct negative societal impacts.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a biologically plausible neural network implementation for online ICA. It is based on a similarily objective with inspiration from the FOBI procedure. The authors derive an online learning rule and interpret the terms in the updates in a synaptically local manner with the addition of a global activity level signaling mechanism. The authors then validate the method in synthetic, speech, and image datasets.
main_review: Overall, the paper is clearly written, the derivations are sound as far as I can tell, and the results show the applicability to synthetic and real datasets. The results are of interest to readers at the intersection of machine learning and computational neuroscience and possibly hardware implementations of neural networks.

Although I have not run it, the provided code looks like it will reproduce the results from the paper.

Most of my comments are related to clarifying certain points in the paper.

- For the references on lines 111-114, it would be helpful to state whether or not the application of Similarity Matching lead to local or online learning rules for any of those models. Are any of the particular choices that lead to a local, online algorithm for BSS generally applicable? Those references share the most conceptual overlap with this manuscript and it would help to have some clarificaltion on how much technical overlap there is across methods/models.
- Since $||y_t||$ is not clearly a local variable, it requires some interpretation in terms of a potential global modulatory component, which the authors provide. One point that could use some discussion is about how to match timescales since $||y_t||$, at least in the current presentation, needs to modulate at the data sampling rate. Does this point towards an interpretation of faster modulating processes from lines 214-219 being a better potential biological counterpart?
- The difficulty of using Oja’s rule for FOBI on lines 95-97 could be more clear, particularly the sentence that starts on line 95 (what does "true output" mean?). Is another way of stating the problem that Oja’s rule could be applied to $z$ after it is computed, but the goal is to learn a $W$ that applies to $h$?
- Are the results presented in the main text for the offline or online version of the algorithm?

- Related to limitation 1 below, I would have expected the online algorithms to have some amount of “burn-in” time where the errors were higher before they become adapted to the statistics of the dataset, but this is not apparent in the figures. Is this masked by the trial shuffling for training and then unshuffling for display? Can you show (in the appendix) how fast the online variant’s error converges toward the offline solution?
- How fast does the iteration in Eq 16 converge?

- In the second paragraph of the introduction, it might make sense to have a sentence about biologically plausible networks that learn other models similar to ICA, for instance refs [46], [48], and 
    - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007692
    - https://www.sciencedirect.com/science/article/pii/S0896627317304178
    - https://www.jneurosci.org/content/33/13/5475.short
    - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002250

Results on data
- For Fig 2 A, I think it would be more clear to group the 4 mixtures on the left and the 4 recovered sources on the right. Alternating them gives the impression that there is some connection between mixture 1 and source 1, mixture 2 and source 2, etc.

Minor comments

- 280: reconstruct -> reconstruction
- Fig 2 caption: second sentence needs correction
limitations_and_societal_impact: 1. One algorithmic mismatch is the desire to have “online” be a requirement for biologically plausibility, but then time-shuffling the training examples to remove temporal correlations (if I’m understanding Appendix B correctly). Ideally, an online biologically inspired learning algorithm should be applicable to streaming data which may have temporal correlations. Achieving both an online + synaptically local network and learning from correlated timeseries may be beyond the scope of the paper, but it should be addressed at least in the Discussion as a limitation/future direction.
review_title: null

Meta-review: Dear authors,

reviewers have reached a clear positive consensus on your work and I
therefore endorse your paper for acceptance.

Best regards
The AC
Decision: Accept (Spotlight)