Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper evaluates previous work on biologically plausible DNNs in the setting of continual learning (CL). Namely, they evaluate ideas around Dale’s principle, Active Dendrites, heterogenous dropout, Hebbian learning, synaptic consolidation and experience replay. They present experimental evidence that these ideas can individually improve the performance on CL, or be combined to achieve greater improvement.
strength_and_weaknesses: + This paper provided a useful list of biologically-inspired modifications to DNNs.
+ I find the main idea interesting (investigating such modifications’ usefulness to CL).
- The paper is a compilation of already published work.
- Some of these insights (s.a. dropout being useful for CL) can be found in literature.
- The experiments are exclusively done on MNIST-based datasets. While they provide evidence to support the claims of this paper, it would be useful to understand whether this translates into other image domains.
clarity,_quality,_novelty_and_reproducibility: I found the discussion around the properties of biological neural networks a bit confusing. Concretely, I don’t think that the terminology was clearly described.
Novelty: I do not think that the ideas expressed in this paper are novel. A “related work” section would be a great way to set the paper apart from similar insights expressed in CL literature, and point out how the ideas of this paper are different.

summary_of_the_review: While the paper presents an interesting discussion, it reuses ideas from other work and presents limited experimental evidence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work the authors study the individual and combined effect of a range of biologically inspired mechanisms on continual learning. 
strength_and_weaknesses: Strengths
- Interesting and important research for both AI and neuroscience
- Paper is well written and all approaches are clear and well described 

Weaknesses
- No floor (e.g normal training) and ceiling (e.g. full replay) controls provided in the experiments.
- Experiments limited to variants of MNIST. Unclear if active dendrites approach & prototypes/context will be as effective with datasets without an exemplar structure, like fashion mnist. 
- Unclear how the difference mechanisms interact with each other, but possibly beyond the scope of a conference paper. 
- Why do active dendrites fail with seq-mnist?
- Is there biological evidence for context dependent dendritic processing of eq. (4). 
- Heterogeneous dropout does not seem to be biologically motivated. 
- Novelty appears limited to combining existing approaches, the authors could more clearly describe their specific novel contributions. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and of good quality. Novelty appears somewhat limited. 

summary_of_the_review: Overall I think this is an important and interesting study, though interactions between mechanisms could be explored better and some controls are missing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors claim that including biological features in artificial neural networks can improve their performance on tasks such as continual learning. The authors propose to include four different biological features into their models and evaluate their relevance: 1) segregation of neurons into populations of excitatory and inhibitory neurons as in Dale's principle. 2) Dendritic-inspired structures as modulators and sparsifiers. 3) Hebbian plasticity rules. 4) Experience replay and regularization. 

The different scenarios are evaluated based on variations of the famous MNIST dataset, showing the influence of the different parameters when considered together or separately. The authors claim that the results obtained offer compelling evidence. 
strength_and_weaknesses: The paper addresses an important question on the importance of biological features for guiding the design principle of artificial neural networks. 

However, it is unclear how we can evaluate the importance of such features with the currently limited evaluation. How does it generalize to other datasets? What about synthetic datasets, where one can control for the different structures of the data that the features are supposed to be useful for?  

The influence of the different parameters is primarily measured in accuracy, but how is that the suitable proxy for measuring the impact of these features? Why are those features even relevant in this context? 

What is the rationale for combining Dale's principle, pyramidal-like cells, and Hebbian principles? Pyramidal cells are arguably not driven by the Hebbian principle. How is sparsity enforced without interneurons?  
The biological plausibility of those features is mostly in name but is vaguely plausible in their current implementation; as such, it cannot be one of the paper's main claims.
More rigor in how these concepts are handled would be greatly appreciated. 

clarity,_quality,_novelty_and_reproducibility: The paper is relatively well written. The motivations and rigor expected are lacking but various things are suggested in order to possibly improve the current submission. 

The novelties are minor, it is rather incremental work. 
summary_of_the_review: This paper addresses the important question of the influence of biological inspiration on the design of artificial neural networks. 
The paper falls short however in how the influence of the different features is evaluated and motivated. 
It is a bit of a patchwork of biological components rather than a cohesive model. 

I greatly encourage the author to further this research as it can provide a great alternative to the blind study of artificial neural networks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors introduce a biologically inspired neural network architecture that combines a variety of learning and processing mechanisms, with the goal of studying whether the performance of artificial systems might be improved by increasing their computational realism. In particular, the authors explore the implementation of Dale’s principle, Hebbian updates, sparse coding, heterogeneous dropout, synaptic consolidation, and experience replay. The proposed model is tested in continual learning (CL) tasks derived from the popular MNIST benchmark, and results suggest significant performance gains.
strength_and_weaknesses: One of the strengths of this paper is that it combines a remarkable number of recent ideas and biological processing mechanisms into a unified architecture. The proposed model is also tested in an incremental fashion, which allows to better understand the contribution of each mechanism / biological feature in final performance.

A major weakness is the adoption of fairly “easy” tasks, since performance is almost at ceiling in most task variants (the only challenging one seems to be the Seq-MNIST). This might not allow to fully appreciate the gain associated with each model variants (e.g., Hebbian Update in Tab. 1). The latter issue is also amplified by the small number of runs (n=3) for each condition: increasing this number to at least 5 or 10 would allow to better clarify the actual gains, at least qualitatively; increasing it even more (maybe in some representative cases) would further allow to verify the gains through statistical testing. It would also be interesting to consider the full “Bio-ANN” model without Hebbian Update, to check whether the overall performance gain in the combined model really depends on the inclusion of such principle. Moreover, it might be useful to compare the proposed model with some other state-of-the-art approaches in continual learning to better highlight its potential (and possibly its limits).
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is written in a very clear way. The overall framework is properly motivated, the related work is (to the best of my knowledge) appropriately referenced and the results are presented in a comprehensible way.

Quality: The results are interesting, and the methods seem appropriate. As stated above, I would suggest increasing the number of runs for each condition to better establish the statistical significance of the performance gains, and possibly include the comparison with an alternative CL approach and/or a more challenging CL task to more robustly validate the model.

Novelty: The proposed architecture is original since it combines many recent (or even very-recent) ideas into a unified framework.

Reproducibility: I think the results are reproducible.
summary_of_the_review: Overall, my judgment of this paper is positive. The research scope of this paper is broad enough to be of interest to the audience at ICLR and stimulate further interdisciplinary research work. At the same time, I believe that further simulations are required to make the paper stronger.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the impact of adding various biological details to artificial neural networks, including active dendrites, Dale's principle, Hebbian plasticity, and others. The authors demonstrate that when all of these components are added to a multi-layer perceptron it can improve the performance relative to a model using only one or a few of these mechanisms, particularly for continual learning.

This was a borderline case. Some reviewers were fairly positive, noting the clarity of the paper and thoroughness of the experiments. But, other reviewers were concerned that the paper sits in an uncomfortable space where its contributions to either neuroscience or ML are unclear.

After an AC-reviewer meeting (see more below) it was decided that the goals of this paper are not sufficiently targeted. Specifically, if the goal is to improve the state-of-the-art (SotA) in ML for continual learning, then the paper doesn't quite deliver, as it doesn't compare to modern ML techniques nor in more SotA architectures. In contrast, if the goal is to understand why these mechanisms may have evolved in the brain, then there is insufficient analysis (theoretical or empirical) to really help us to understand why these mechanisms have the effects that they do. Thus, though the paper has some promising aspects, it was agreed that it requires more work to focus on a more explicit goal, whether improving ML or increasing understanding of the brain. Given these considerations, and the average score of 5 post rebuttal, a reject decision was reached.
justification_for_why_not_higher_score: Though this paper is well-written and thorough, it arguably should have been more targeted in its goals. Either it should have focused on improving ML with biological insights, or helping us to understand why these mechanisms exist in the brain. Instead, it tried to do both simultaneously, in a manner that arguably failed to do either fully.
Decision: Reject