Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper asks the question of why neural systems often develop a factorized or disentangled representation, i.e. a representation which aligns its variables/latent units with a meaningful factorization of the underlying problem structure. The authors use a combination of theory and simulations to show that biological constraints of non-negative and energy-efficiency together gives rise to disentangled representations. Finally taking the specialized cells in hippocampus like grid cells and object vector cells as a biological case in point, the authors simulate spatial processing in ANNs and demonstrate that biological constraints lead to disentangled codes for space and objects when these factors are independent, but the same constraints lead to entangled codes when these factors do not vary independently in a task, providing a potential explanation (specifically) for why grid cells warp their receptive fields and (more generally) for when and why disentangled codes arise in the brain. 
strength_and_weaknesses: Strengths: 
- I think the paper is exceptionally written and conveys the ideas very well. 
- The paper is a nice mix of theory and experiments along with a biological case simulation. The authors also present an intuition accompanying the theorem which nicely illustrates the logic.
- Instead of simply comparing representations at the population level which has been the dominant approach in neuroAI, the paper compares representations at the level of axes, i.e. why the brain and ANNs often develop privileged axes. This is an intriguing question that has received little attention in the field so far and I laud the authors for tackling this with clarity. 

Weaknesses: 
- I think the justification behind non-negativity (of firing rates) as a biological constraint needs some further discussion in the paper. It is known that the firing activity of single neurons can be suppressed (from baseline rates) and activity is always going to be measured with respect to some reference signal/rate. The constraints the authors frame as ‘biological constraints’ are both routinely employed in modern CNNs through ReLU and l1 regularization on activity/weights. In that context, I found the use of the phrase ‘biological constraints’ to be a bit overblown. 
- The authors claim their results provide a potential explanation for concept cells (i.e. grandmother cells) but if energy considerations favor disentangled codes, we would expect to find selective codes for every single concept. Clearly, allowing specialized cells for all concepts in complex domains would be impossible as we’d run out of neural hardware. It would be useful if the authors could discuss the trade-offs or speculate about how such trade-offs might be handled by the brain. 
- I understand that space might be a limiting factor but it would be helpful to situate their work within the context of a broader literature on disentangling in machines (besides beta-VAE). There is plenty evidence of disentangling in machines as an emergent phenomenon (not baked in) for e.g. the work of David Bau et al. who found interpretable neurons in vision models, multimodal neurons in CLIP (https://distill.pub/2021/multimodal-neurons/), other work by on superposition/ privileged axes (https://transformer-circuits.pub/2022/toy_model/index.html) that also discusses the role of non-negativity constraints etc. There is also other work on understanding specialization in DNNs that could be relevant: https://openreview.net/forum?id=HreeeJvkue9. 

clarity,_quality,_novelty_and_reproducibility: I found this work to be of very high-quality and consider this to be a novel and original contribution to the field. 
summary_of_the_review: I think the question of why neural systems have a factorised code has received little attention from a computational perspective and this paper is an important step towards filling this gap. For the AI community, understanding principles that promote disentanglement is important as they make generalization/transfer to new tasks easier. This paper illustrates important principles for achieving disentanglement in models (although they are already often implements in ANNs albeit without a true understanding of its consequences).  Overall, I think this is a very interesting paper that makes a novel contribution and would appeal to both the neuroscience and AI community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: I am very impressed by the authors' thorough replies to all the concerns I raised. I can see that a lot of thought and work has gone into improving the papers' clarity and claims. I am raising my score accordingly.

The authors show that non-negativity of neural responses together with an energy constraint encourage disentangled representations. They provide a mathematical proof for this conjecture in two linear settings: linear neural responses, and linear generative models. They demonstrate that this work on linear data and, apparently, also on nonlinearly mixed data. Finally, the show how these constraints encourage distinct cell types in a model of spatial navigation.
strength_and_weaknesses: Strengths: The paper touches on many interesting and highly relevant topics, including: the relationship between non-negative latent variable models (e.g. NMF) and source separation (ICA); nonlinear source recovery; and the reason for the existence of distinct cell types. The mathematical proof and the linear source recovery simulations appear to be valid.

Weaknesses: It is rather frustrating to see how all these fascinating research questions are dealt with in a short inconclusive manner. The relation between non-negativity and source separation alone would provide more substance than could easily fit into a ICLR conference paper. Ideally, this would then also include a much more thorough literature research and connection to existing source separation and (non-negative) tensor factorisation (like NMF) methods since the early 90’s, as well as a number of additional control studies (changing the number of factors, their distributions, testing violations of different model assumptions, benchmarking against other methods like sparse coding etc.). The other two subjects nonlinear ICA (disentanglement) and cell type emergence are also extremely interesting and deserve a much more thorough treatment.
clarity,_quality,_novelty_and_reproducibility: Generally, the paper lacks a lot of technical details. It is often not clear what the task is, or what the model is that is being evaluated, or what the evaluation metric is (see below). 

Random Points:

Please use the more common term ‘disentanglement’ (rather than ‘disentangling’) in the paper, this would at least align with the ML literature. Ideally, go beyond the buzzwords used in ML and relate to the proper terminology from source recovery and ICA.

The last sentence of the abstract is very confusing and I am not sure what the authors are trying to say.

The second paragraph of the introduction is a collection of random facts; the term disentanglement is not defined; what is the message of this paragraph, please clarify.

3rd paragraph: this (https://arxiv.org/pdf/1206.5538) would be the standard reference for sources of variation; this (https://arxiv.org/pdf/2107.08221.pdf&lt;/p&gt;) challenges the idea that disentangled representations generalise better; What is the evidence for the trade-off mentioned in the last sentence?

1.1. Reference for Oja rule? Also, there is a whole line of research missing here. Barlow already postulated that neurons perform redundancy reduction, i.e., minimising mutual information between their representations, i.e., finding independent components. BTW, that would also take care of the multiple neurons representing the same factor issue seen here.
‘While it has beeen shown’ +’that’

D. in brains: here are other references about disentanglement in brains (https://elifesciences.org/articles/43625.pdf) and (https://www.science.org/doi/full/10.1126/sciadv.aax5979). The second to last sentence needs more explanation.

2: what is ‘neural disentangling’? What are a and C? Why is C the same for all neurons? Why minimise the square of the expectation and not the expectation of the square or the variance? Why write |M_jk||M_jl| and not just state that every row of M has exactly one non-zero entry?

What is the dimensionality of x?

Theorem 2: the zero error assumption seems rather unrealistic?

Fig 2: What is the scale of the MI matrices? The whole plot and scripts are too small; Define MIR; What is the noise ceiling in D?

3: 2nd paragraph it is not clear what the task is or what setting is being studied here. One problem with the proposed measure of disentanglement is that for factors a,b and neural response z: I(z;a) and I(z;b) may be low, but I(z;(a,b)) may be high – For instance, if there is a spiral-shaped function with bases a and b and height z – the same problem of pairwise vs. Multi-information measures exists also for the MIG metric.

Page 4 bottom: Again, what is the setting? What is dim(y)?

Fig3: b) what is the noise ceiling?

Disentangling in Deep Nets: how is the data generated, what is the model? Please provide more details.

Disentangling in unsupervised deep nets: You cite Locatello et al. 2019; So you are aware of the fact that this setting is equivalent to nonlinear ICA and that, theoretically, this should not work. Please comment on this. A similar construction as in the original Hyvarinen & Pajunen, 1999 (nonlinear ICA is not identifiable) result can be extended to cover the non-negative setting here. Are the authors claiming that their theory extends to the nonlinear mixing case? If not, please comment why, empirically, it seems to be working to some extent. Maybe because of (https://openaccess.thecvf.com/content_CVPR_2019/papers/Rolinek_Variational_Autoencoders_Pursue_PCA_Directions_by_Accident_CVPR_2019_paper.pdf)? Which could be broken by (http://proceedings.mlr.press/v139/zietlow21a/zietlow21a.pdf)?

‘VAE loss already’ + ’includes’

If you evaluate your method on the dislib benchmark, then please provide proper comparisons against other methods, all metrics and with a sufficient number of seeds (see Locatello et al, 2019).

Fig5: b) what is beta_weight? What is Goldilocks?

A factorised task for rodents: second to last sentence, I don’t see how this falls out from the theory?

The use of the term path integration is misleading at the beginning of the next paragraph.

Fig6: explain the Real OVCS subplot; C) the grids do not appear to be hexagonal or split into modules of different spatial resolution like real GCs? Explain subplots D; explain the spatial correlation and the subplots in H;

Equation 3 needs much more explantation, what is being shown here?

The sum over loss terms in equation 4 is astonishing, one might wonder whether, with that many degrees of freedom, the model can be forced to produce all sorts of results – put differently, if all these hyperparameters can be adjusted, is it informative if, in the end, a setting is found that leads to the presented results?

‘Some cells change a lot, some cells don’t change […] thus, the representation has two disentangled modules’ – what is the quantitative measure for this claim. Honestly, this is frustrating, these are very interesting questions that seem to be dealt with in a hand waivy manner.

Last paragraph section 4: please clarify the distinction and usage of the terms stereotyped behaviour and the concepts of independent factors used so far.
summary_of_the_review: This paper addresses some very interesting questions. They provide an interesting mathematical derivation that could provide a substantial contribution at the intersection of ICA and non-negative factorisation (NMF, sparse coding etc.) literature. However, this would need a much more extensive treatment with more details. I think there are big questions related to the nonlinear setting (see Locatello et al., 2019). The arguments on cell types are interesting. However, the RNN model has a lot of degrees of freedom, the experimental results are unclear and one might wonder if this theory of efficient coding and sources recovery (which needs more links to existing literature) would also apply to, e.g., visual neurons.

I strongly encourage the authors to split this up into multiple papers and do a more thorough scientific investigation of the different component claims: 1) ICA and non-negativity; 2) non-negativity as inductive bias in nonlinear ICA?; 3) cell types and non-negativity leading to separated components. Each of these questions is very important and interesting and it would be great to give them a proper treatment that creates new scientific insights, ideally with higher SNR than the average ML paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript introduced a bio-inspired representation learning that guarantees disentanglement in linear network, under some assumptions. The authors showed two neurobiological constraints i.e., nonnegativity and minimum energy efficiency (with respect to activity and weights), lead to linear factorization, under iid assumption of true factors. Additionally, the authors utilized the proposed biological constraints to interpret cell types which are functionally distinct. The authors used multiple simulated scenarios to demonstrate the performance of the proposed approach for linear and non-linear networks, under supervised and unsupervised fashion. Additionally, they reported their model disentanglement performance in comparison with beta-VAE for a few examples of benchmark datasets including Shape3D, dSprites and a rodent navigation task.

**Update after rebuttal:** I have read the author's rebuttal and the latest revision. The authors have satisfactorily addressed most of questions/concerns and improved the manuscript by providing more details and clarification. After reading the rebuttal and the response to the other reviewers, I'm happy to raise my score to help the paper getting accepted.
strength_and_weaknesses: Strengths: 

- The paper is addressing a challenging and important problem, learning disentangled and interpretable latent factors, from a neurobiological perspective, which in my opinion is quite interesting and novel. The authors provide a simple set of constraints which lead to a disentangled representation which are quite valuable and potentially could expand the findings of the previous disentanglement studies.

- The main claim of the manuscript is theoretically and empirically justified. 

- The simulation studies illustrate that under iid the proposed metric reveal the disentanglement decrement under entanglement attacks, while the comparable measures fail to present it.

- The authors provide multiple experimental settings to evalute the proposed method. 


Limitations: 

- Although the paper has attempted to make a bridge between neuroscientific findings and the representation learning field, I think the proposed objective is not well-connected to the other studies in this field, except for beta-VAE. It is not clear how the proposed neurobiological constraints improves the already disentangled representations provided by other approaches like factor VAE, weak supervision method (Locatello, ICML, ICLR 2020), or satisfying local isometry (Horan, NeurIPS 2021 ), all of which do not necessarily rely on decorrelation. It is not clear how these neurobiological constraints will complete those approaches. I think the authors need to elaborate more on this.

- IID Assumptions: I understand that making iid assumption for the latent (task) factors makes the theoretical justification easier, but in practice, especially in the case of neurons, it is not correct to assume that all neurons follow the same distribution.

- Domain knowledge: one of the difficulties of learning a disentangled representation is the absence of domain-specific knowledge, like the number of independent factors. According to the problem definition in Eq. 1, to obtain disentangled factors, the network requires access to the true number of K, as I have understood. If this is correct, is there any way to utilize the proposed approach to learn the true number of latent factors?

- The rodent navigation task is a very interesting experimental study. However, the problem formulation in Eq. 4 is quite unclear and confusing. 

- There experimental results for benchmark datasets are very limited.

Additional Comment/Questions:

- Is the rodent navigation dataset a new dataset used in this paper? What are the modalities of the given data (measurements)? 
- Can you elaborate more on your findings in Figure 7? How should one compare the results in Figures 6 and 7?

- How would this model perform when K is not matched with the true number of factors? In the case of cell type, how can one interpret the results in terms of the function of each neuron?

- How can the proposed method be used to functionally characterized neurons in the other data modalities, e.g. RNA-seq? 

- The authors may want to move some of the supplementary materials, e.g. sequential path integration to the main text. It is quit unclear what Eq. 4 is formulating. 

clarity,_quality,_novelty_and_reproducibility: Clarity: 

The paper is overall well organized and well written.
Section 4, the rodent navigation task, and the problem definition need some revisions and more clarifications, including Eq. 4, Figure 6 and 7.


Quality: 

The paper is technical with enough theoretical backing. Theorem 1 shows that in the presence of iid latent factors and nonnegativity of neural activities with constant variance, the representations that achieve minimal activity energy also exhibit disentangling. Theorem 2 generalizes the result of the first theorem by considering generative model for neural representations that can exactly predict the observed data, without requiring constant variance.

Novelty and Originality:

The paper is addressing a challenging and important problem. The introduced neurobiological perspective and neural representation modeling are novel and interesting. 

Reproducibility:

I cannot comment on reproducibility. The code is not available. 



summary_of_the_review: I think this is a good submission and the proposed neural representation learning using biological constraints is persuasive. However, I think the authors need to improve section 4 and discuss how these findings are generalizable to the other datasets.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The authors investigate how biological brains could achieve disentangled representations. Through a combination of theory and modeling, they provide ample evidence for two key constraints: (1) nonnegative activities and (2) a constraint on energy expenditure in model/real neurons. The work is exceptionally well explained and builds out from these simple principles to explain results in modeling complex artificial data and rodent experiments. 
strength_and_weaknesses: Strengths:

- Great introduction.

- Love how the authors presented and explained the proof relating non-neg and efficiency to independent task factors.

- The related work brings together the parallel investigations of disentangled factors and the use of nonnegativity to do so (e.g., Lee & Seung, 2000) in Neuroscience and ML.

- Honestly I could go on and on, but this is a brilliant paper. I love how it begins with a simple concept, explains it like I'm five, and then shows that the theory holds in complex situations that are relevant to the large-scale ML and Neuro pursuits of today.


Weaknesses:

- Can you add a github link with your code?
- Make the text bigger in your figures? It's a little hard to make out. That's it.

Questions:

1. Are the colored circles in Figure 2 cartoons or real data?
2. In Fig 2, if I'm understanding this correctly, relu and a non-neg activity constraint yield the same performance. Does the regularization also affect the signs of the model weights?
3. There's no constraint for non-negative weights, right? Just non-negative activities?
4. What’s the explanation for neurons that show mixed selectivity? Is that just a relaxation on the energy constraint?
clarity,_quality,_novelty_and_reproducibility: Beautifully presented paper with adroit insights and contributions to ML and Neuro. The work appears reproducible but there's no code release.
summary_of_the_review: This paper is an absolute accept. Beautifully written, deeply insightful, great lit review, and lays out theory that I believe can inform the development of better models for AI and the design of new neuro experiments. Best paper I've reviewed for a conference in years (ever?). I'm going to be a jerk and give it an 8 for now just because the authors didn't release their code. Assuming they do over the discussion period I'll bump them to 10.
review_title: null

Decision: Accept: notable-top-25%