Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper compares two bioplausible implementations of Mahalanobis whitening based on (anti)hebbian plasticity rules, in either a network with direct connections or a network with interneurons. The core finding is that without interneurons, the convergence times scales linearly, but with interneurons it scales only logarithmic.
strength_and_weaknesses: Strength:
* Nice analytical result with biological interpretation and numerical confirmation. Very clearly written, transparent derivation. 

Weaknesses:
* Not clear how that would be done by a more biologically plausible neuron using spikes.



Minor issues:
"overeparameterized" should be "overparameterized"
"Interestingly, contrast" should be "Interestingly, in contrast to"

clarity,_quality,_novelty_and_reproducibility: * Clarity: 
The structure, wording and math is very clear.

Quality:
The theoretical derivation seems sounds, the numerical test confirm the theory.

Novelty:
The main finding is that interneurons improve the learning timescale of bioplausible implementations of Mahalanobis whitening.
The overall framework is not novel, there have been a series of papers on deep(linear network), but the implications of interneurons on learning speeds is novel to my best knowledge.

Reproducibility: 
The math is fully reproducible using a pencil and paper and linear algebra. To make the numerical experiments reproducible, the authors are kindly encouraged to share their code.

summary_of_the_review: The submission is a nice addition to a series of paper on bioplausible implementations of basic algorithms (PCA, CCA,etc) with a plausible biological interpretation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the learning dynamics of a linear recurrent network with and without interneurons. Theoretical analysis and numerical simulation both suggest the network with interneurons converges faster in learning.
strength_and_weaknesses: The present paper is mathematically sound by theoretically analyzing the learning dynamics. I have no problem with the techniques and the results of the paper. 


### Major questions
(PS: below are some of my questions regarding the results which are not a criticism of the present study.)

- The interneurons in neuroscience studies are inhibitory neurons. For the interneurons in the present model, do authors constrain the polarity of interneurons' weights? I don't find this constraint and I guess no such constraint in the study. Moreover, for the network model with interneurons, there are no direct recurrent connections between y neurons in the network (Fig. 1, right). I am curious how the result will be changed if we include direct recurrent connections.

- The recurrent connections $W$ in the network with interneurons correspond to decompose the recurrent connections $M$ in the network without interneurons, i.e., $M=WW^\top$. This implicitly assumes the connections between primary neurons y and the interneurons z are symmetric. I am happy to see some discussions in releasing this constraint in that such symmetry probably doesn't exist in the brain.
clarity,_quality,_novelty_and_reproducibility: - Clarity:
The paper is well-written and the math derivations are clear.

- Novelty: the learning dynamics of a linear recurrent network should be well-established. The part of a recurrent network with interneurons seems novel.


summary_of_the_review: I have gone through nearly all the math and I am certain that the math derivations are correct.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors investigate the role of inhibitory interneurons in the statistical adaptation of the brain. More specifically the investigate their role in neural networks that perform input whitening. The authors demonstrate that networks with inhibitory interneurons are more resilient to initializations than networks with just direct recurrent connectivity that operate in violation of Dale’s principle.
strength_and_weaknesses: Strengths 
Analytical derivations and numerical simulations convincingly support the claims of the paper. 

Weaknesses 
Dale’s principle is a fundamental dogma in Neuroscience and thus structures that violate it cannot be deemed biologically plausible by our current standards. Comparing the advantages of networks that follow Dale’s principle against ones that don’t could be arguably deemed as little more but an interesting mathematical exercise. The main result of the paper, namely that networks obeying Dale’s principle are more robust to initializations is a bit underwhelming and likely to have very low impact in the broader field of Neuroscience and RNNs.    

clarity,_quality,_novelty_and_reproducibility: Clarity 
The paper is clearly written, but it would benefit from including a small section that highlights the main contributions of the work as well as the broader impact of the results. 

Quality 
The quality of the paper is sound. The claims in the paper are backed by solid analytical derivations and relevant numerical simulations, and I have not come across any obvious errors in them although I have not checked the math in detail.

Novelty 
The finding/proposal that interneurons could make neural networks more robust against initializations is novel, but arguably, of limited relevance. 

Reproducibility 
The authors provide enough information to enable others to reproduce their work. I have not personally attempted to reproduce the key results of the paper.
summary_of_the_review: A solid and interesting paper, however the impact of the findings seems a bit too limited for a broad venue such as the ICLR conference.  Maybe a more specialized venue could be a better fit for this paper. However, I am aware that I may be overlooking some aspects of the potential impact of this paper, and I will be happy to be corrected and will adjust my rating accordingly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a theoretical analysis of the computational benefits of interneuron-mediated recurrent connections by characterizing the synaptic change dynamics while learning to whiten an input. Specifically, the authors provide analytical solutions to the dynamics of a recurrent neural network without any pointwise non-linearity when the recurrent connections are direct and when they are mediated by a layer of interneurons. They motivate the problem from a computational neuroscience and neuromorphic setting and demonstrate that interneurons enable faster synaptic plasticity dynamics, thus suggesting that it is beneficial to have an intermediate layer of interneurons in biological recurrent neural networks for faster learning. In this work, the cost function is assumed to be the whitening objective, i.e. the activity in the recurrently connected neurons are pushed towards having an Identity covariance matrix.
Although the results and insights are interesting, the work builds on certain strong assumptions that limit its utility to current studies in computational neuroscience or machine learning.
strength_and_weaknesses: Strengths:
1. The paper presents a strong theoretical analysis of synaptic change dynamics under optimization of the whitening objective. This analysis would be useful and is pertinent to current advances in self-supervised deep learning, specifically the BarlowTwins or VIC-Reg objectives (Zbontar _et al._ 2021, Bardes _et al._ 2021).
2. Through numerical simulations, the authors demonstrate that some of the assumptions in their analytical results, albeit necessary for deriving the closed form expressions are not overbearing in practice. These results suggest that the inferences hold even if commonly used initialization strategies in machine learning are adopted. 

Weakness:

I have some concerns about the presentation, but I believe they can be fixed and I have described them in detail under "Clarity, Quality, Novelty and Reproducibility". 

1. My main concern is from the computatinal neuroscience perspective. The authors present Dale's law as a motivating factor to understand the interneuron-mediated recurrent connections. However, it is not clear if the sign constraint is applied on the recurrent connectivity matrix or if the simulations used projected gradient descent to ensure that the weights were always all +ve. With the negative sign incorporated in the network formulation, the interneurons are supposed to be inhibitory but if the weights are allowed to be both +ve or -ve, it doesn't respect the Dale's law anymore. 
2. The overparameterization narrative assumes that there are more interneurons than pyramidal neurons ($ k \geq n$), which is generally not observed in biological neural circuits. Instead it is the other way around. Could the authors comment on this and what it means for their results and inferences? Additonally, the constraint on weights being tied to and from the interneuron population, i.e. the requirement for $W$ and $W^T$, is biologically unrealistic. Could the authors comment on whether this is necessary?
3. The whitening objective provides a nice analytical setup wherein the authors converge to a closed form solution. Although the difference in dynamics for direct vs interneuron-mediated connections is presented in the paper, it is unclear how these learned representations are different, if at all. This understanding will be important for both computational neuroscience and machine learning, wherein an important question is do interneurons provide a computation benefit in realistic tasks. In other words, are interneuron-mediated recurrent networks better able to learn certain tasks because they can whiten the inputs quicker? Demonstrating this in a toy task would also provide useful evidence in favour of such an architecture. If not, it raises a question whether whitening is a reasonable objective to consider when learning representations from data. 


clarity,_quality,_novelty_and_reproducibility: Writing clarity/quality:
1. Firstly, I would suggest the authors to reformulate their recurrent neural network setup to be more aligned with the convention in machine learning, given the audience of ICLR. Specifically, it is unclear how the information propagation happens in the network from the current presentation. From my understanding, the network can be thought of as receiving X as input, the input to hidden weight matrix being identity and hidden to hidden weight matrix being $-M$ (or $-WW^T$). This would simplify the neural activity dynamics to be governed by the incoming current to the neurons, $X - MY$ (or $X- WW^TY$). 
2. If the above is true, can you comment on how things would change if the input to hidden weights are non-identity and/or learnable?

The work seems to target an interesting question and although there are other work that aim to understand the computational benefit of interneurons and/or inhibitory connections, I am not aware of any other work that characterized the learning dynamics in recurrent networks. So, this work presents a novel perspective. 
Unfortunately, I didn't have time to try and replicate the results from the information provided by the authors. So I am unable to comment on the reproducibility. 
summary_of_the_review: Thir work presents an interesting perspective about interneuron-mediated recurrent connections, but it uses certain biologically implausible assumptions. Moreover, the definition of recurrent network used here is slightly different from the canonical definition used by the machine learning community and given the audience of ICLR, I feel this work might not be pertinent to a broad audience. Finally, despite the interesting analytical results and corresponding empirical validation, it is unclear how the benefits in dynamics translates to computational benefits with respect to task performance or learning ability. Therefore, this work would need significant changes for it to be presentable to the ICLR community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper compares two different architectures of linear recurrent neural networks for data whitening, one that has direct recurrent connections, and another where recurrent interactions are mediated by interneurons. Through analysis of synaptic dynamics and simulations the authors show that the network with interneurons can converge more rapidly regardless of initialization. They provide an account of these results via the concept of over-parameterization.

The reviewers were generally positive about this paper. There was consensus that the results were technically strong, novel, and of potential interest to some readers.

There were some concerns regarding biological relevance (e.g. lack of Dale's principle being respected), how widely relevant the results are, and whether the rapid whitening leads to more optimal representations. Following rebuttals, most of the reviewers were satisfied, and the final average score was 6.75. Given this, and that there is agreement that the paper is technically sound, a decision of accept was reached.
justification_for_why_not_higher_score: Though this paper is technically sound, the reviewers did note that its potential impact is somewhat limited. As such, a spotlight is inappropriate.
justification_for_why_not_lower_score: With a technically correct paper that has an average score of 6.75, I believe that a reject would be inappropriate.
Decision: Accept: poster