Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors construct an architecture based on sparse-distributed memory (SDM) and multi-layer perceptrons and show that this architecture is naturally robust to catastrophic forgetting. This architecture uses sparse activations of neurons (top-K) with a couple of tricks to improve trainability. One one continual learning task (CIFAR-10), the authors show that this produces very good performance, esp. when combined with EWC.
strength_and_weaknesses: ## Strengths

- The authors clearly demonstrate that SDM in the specific form they use is suited for continual learning, with good ablation experiments
- Detailed experiments on the one benchmark that is chosen.
- The authors contribute a few tricks for training SDM-based architectures that are novel.
- Methods are clearly explained.

## Weaknesses

- Evaluation done only on one benchmark.
- Not clear if biological plausibility is a goal, and if so, how it is achieved.
- Differences from (Shen et al. 2021) not clear.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the methods are clearly explained. The overall quality of the paper is high.

The paper seems novel to my knowledge, esp. with the specific set of tricks the authors use to train the SDM+MLP architecture. But relation to (Shen et al. 2021) needs to be discussed in more detail.

The paper seems to have sufficient experimental details, esp. for the training, and hence looks to be reproducible.
summary_of_the_review: Overall, it is a well-written high-quality paper with very interesting and relevant results for continual learning. The paper's biggest weakness is having results (albeit detailed) on only one benchmark. The paper needs comparisons on a more diverse set of benchmarks, and is the primary reason for my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper interprets Spare Distributed Memory (SDM) as a continual learner. Analogous to a one-hidden-layer MLP, the SDM is modified to support continual learning. There are several training tricks, such as the choice of optimizer and the training procedure. The most notable change is to replace the binarization activation in SDM with the top-k function, where k is annealed over training time. The method shows improved performance in the class-incremental setting with  CIFAR10 dataset.

strength_and_weaknesses: ### Strength
- The paper gives a new insight: SDM can support continual learning, mainly based on its sparsity nature. 
- The paper has done a non-trivial task to make SDM work for actual data, which involves many tips and tricks in training SDM. 
### Weakness
- The presentation is hard to follow. The main text lacks details to understand the method. The paper is not self-contained and often requires references to Appendix, which is only optional for the reviewing process.
- Too many modifications to SDM make the final method no more SDM, which depreciates the paper's central message. Also, the amount of tuning hyperparameters and calibrating training procedures is enormous, making it hard to apply to a different CL data or task. 
- The experiment is limited. The proposed method (combined with EWC) is only better than FlyModel in CIFAR10, and underperforms in CIFAR100 and MNIST

### Detailed comments and questions:
- Eq. 3, how do you write to $X_v$? If you use SDM's write, what are the $P_a$ and $P_v$ in the case of CIFAR10 data?
- The paper claims to fix the issue of random $X_a$ with top-K activation. However, Eq. 4 only presents the activation function, which affects the output $a$. How does it help $X_a$ model real-world data?
- Additional modification: (v) How do you use backpropagation in SDM? (to update $X_v$, $X_a$?)  
- Please consider adding an algorithm to the main text to clarify how your proposed components work together.
- Please consider more datasets to validate the performance of your method: CelebA, FahsionMNIST, .... Given the current result; it is hard to say your method is more effective than FlyModel
- Why don't you make stronger baselines by combining methods just as you did with SDM? For example, FlyModel+EWC or use more baselines such as EWC, SI, HAT
- Check the section numbering. It should start from 1. Introduction ...
clarity,_quality,_novelty_and_reproducibility: The paper is not self-contained, which impairs its clarity significantly. The quality is good given the amount of investigation and study presented in the main text and appendix. The finding is interesting and new. However, as mentioned in the Weakness, the final method looks so different from the original SDM. Reproducibility is challenging if someone wants to apply the method to a new dataset. It will require a lot of tuning. 
summary_of_the_review: The paper provides an interesting insight into SDM as a continual learner. The method, however, is over-complicated with many components and tuning efforts. The experimental result is weak.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a number of modifications to the standard Multi-Layered Perceptron (MLP) to avoid catastrophic forgetting in continuous learning tasks. The proposed modifications are motivated as biologically plausible and include the use of sparse distributed memory, a Top-K activation function, no bias terms, and L2 normalization and non-negativity constraints on weights and data. These features are studied in isolation and combined with other approaches, like Elastic Weight Consolidation (EWC). In addition, the paper proposes two training techniques: i) an implementation inspired on the "GABA Switch" to avoid "dead neurons", showing this is equivalent to annealing number of active neurons (K in the Top-K activation function) when the non-negativity constraint is imposed; and ii) using an optimizer without momentum to avoid updating neurons that are not in the Top-K set and another potential exploding gradients issue. The authors consider a three stages during training: i) first, pretraining a ConvMixer on ImageNet; ii) pretraining the SDM using the pretrained embedding; iii) continuous learning of the pretrained SDM module with the pretrained image embedding. Experiments show the proposed modifications to the MLP, combined with EWC, provide SOTA results for CIFAR 10 in the challenging class incremental setting, and achieves second better performance for CIFAR 100 and Split MNIST. 

strength_and_weaknesses: # Strengths
The paper aims to provide a biologically plausible extension of MLP that naturally results in continuous learning capabilities. This is an important and usually overlooked research area. 

The connection between SDM, MLP and Transformer is very interesting and it is developed in further detail.

The numerical results are very promising. Although the results are not SOTA for CIFAR100, they are still competitive and outperform other baselines. 

More importantly, the extensive but practical explanations behind their design decisions, the insightful ablation studies, and the suggested directions for future work, will likely have an impact in the community and will foster future research.

# Weaknesses
Simulation results are promising. However, it would be reassuring to see a common trend with other data modalities, like text. For example, the authors could use a pretrained modern Transformer architecture (e.g., DistilBERT) and evaluate continuous learning while fine tuning to different tasks.

The motivation of a single layer MLP is not very clear. It would be interesting to see whether the results will hold for a multilayer MLP.

The explanation of the three stages training regime in the main text is not very clear (probably too concise due to lack of space). The authors could clearly enumerate the three steps, explaining a ConvMixer is used for pretraining from the beginning, and making clear what SDM Module refers to their SDM-inspired MLP variant (maybe coining their model something like "SDM-MLP" could prevent the risk of a reader thinking that SDM and MLP are two different models).

The ablation study on training directly on image pixels would be more relevant if they were done by tuning the loss coefficient and the $\beta$ parameter for training directly on pixels.
clarity,_quality,_novelty_and_reproducibility: # Originality
The paper proposes multiple biologically plausible extensions to MLP that result in a strong continuous learning architecture. This is in contrast with most previous approaches that relied on artefacts to induce sparse networks and specialise them to different tasks.

The authors provide novel insights on how the proposed extensions are able to tackle the "dead neurons" problem. Moreover, the authors identify "state momentum" as a problem that has been missed by previous works. 

The MLP formulation of SDM shades new light on its connection with the attention mechanism from Transformer architectures.


# Quality
All the architectural design decisions are discussed in detail, including extensive related work. 
Experiments are well designed with promising results. 
Limitations of the approach have been identified and discussed.

# Clarity
The paper bridges the gap between deep learning and computational neuroscience. This is not an easy task, but I think the authors did a fantastic job, making the neuroscience concepts accessible to the deep learning community.

# Reproducibility
The authors have made their code available, which together with the extensive details in the main text and appendixes make me confident the results are fully reproducible.


summary_of_the_review: This is a solid work, well written, full of insights, with promising results, and very relevant. I think it will foster future research on both SDM and continuous learning, and has the potential to help the representation learning community to be more aware of the potential of using neuroscience to improve standard deep learning architectures.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is concerned with continual learning, and it is interested in a class of biologically plausible models to overcome catastrophic forgetting. Specifically, the authors focus on the Sparse Distributed Memory (SDM) model, a long standing memory model in computer science. The paper highlights a parallelism between SDMs and a single-hidden-layer Multi Layer Perceptron (MLP) and introduce some tweaks to the latter for empowering continual learning capabilities. Specifically, the authors employ in the hidden-layer a top-K activation function, resembling the sparse read operation in SDMs. Since its naive application introduces the problem of dead neurons, an annhealing strategy is introduced for reducing k (the number of active neurons) throughout the optimization procedure. The authors further notice how momentum-based optimizers are problematic in continual learning tasks, and simply advice not to use them. Experimental results are presented on Split-CIFAR-10 against a number of baselines.
strength_and_weaknesses: - +The paper overall well written, one read is sufficient to the reader to grasp the main ideas and contributions.
- +The authors remarkably place the proposed model within the literature, with a lot of references to both neuroscience and machine learning prior works.
- +The authors present a huge appendix that contains a lot of material useful for a deeper understanding of their research.
- -The motivation that the authors highlight for the employment of SDMs is only biological plausibility. Although I acknowledge that as a good trait, the text does not convey the reason why, technically, the proposed model avoids catastrophic forgetting. My personal interpretation is that, given that the Top-K activation functions create sparse activation patterns, the corresponding back-propagated gradient is also sparse, and less prone to overwrite important knowledge learned during prior tasks. The authors should provide more intuitive explanations about why the proposed model is less prone to forgetting in technical terms, beyond biologically-inspired considerations.
- -The paper describes how both read and write operations are performed in a SDM. However, the parallelism between SDM and a one-hidden-layer MLP is drawn only in the case of read operations (Eq. 3 and Eq. 2). How does the writing mechanism intervene in the authors model? Is the gradient-based learning of the MLP parameters considered the writing algorithm? If so, gradient-based updates do not conform with the writing mechanism for SDMs (Eq. 1). Can we still consider the model a SDM in that case? And if so, why?
- -The main experiment on CIFAR10 presents encouraging results, where the proposed SDM trained incrementally learns better than other models that specifically intervene to reduce catastrophic forgetting. However, the setting is quite arbitrary, as the authors use pretrained imagenet embeddings and an SDM on top of that. This raises two concerns. i) given that imagenet embeddings already separate CIFAR10 classes (as the cifar10 class space is a subset of the imagenet class space), is this an interesting learning setting? ii) are all compared methods in Tab. 1 using the same protocol? In my opinion, the case where *both* the embeddings and the classifiers need to be learned is the most interesting scenario for studying continual learning problems. Can the proposed models be extended to arbitrary backbones (e.g. ResNets), where every pair of consecutive convolutional layers can be considered as a SDM?
- -Table 1 would benefit a oracle where all classes are trained jointly, to put the results into perspective.
- -In Table 2, different optimizers showcase very different performances on the task. Are all baselines in Tab. 1 comparable, meaning that they all use SGD without momentum as an optimizer?
clarity,_quality,_novelty_and_reproducibility: - C: the paper is fairly clear and understandable at a first read. It involves a lot of parallelism with neuroscience terms and literature, that might not be easy to follow for all readers. However, the overall model is explained clearly.
- Q: the research seems conducted with good quality, in terms of literature review, technical formulation and experimental validation. Some issues in the latter point, as expressed above, emerge with the pretraining protocol used on CIFAR-10.
- N: beyond the whole biologically inspired motivation, the technical novelty of the paper reduces to the use of a top-k activation function in MLP hidden layers. In this perspective, the novelty is quite limited. Other contributions such as the annhealing of the top-k and the use of SGD without momentum can be considered tricks and implementation details, that don't add to the technical contribution of the paper.
- R: the authors illustrate in the paper and in the appendix all implementation details and hyperparameters needed to reproduce their work. Moreover, they release an implementation of their model in an anonymous repository, which is remarkable.
summary_of_the_review: In my opinion the novelty of the work is limited for publication at ICLR. The authors reach remarkable performances on a custom task starting from pretrained embeddings, and I am lukewarm about how the model would perform in end-to-end continual learning (i.e. when representations are trained continually as well). The motivation of the work is purely biological, and there is limited discussion about why those ideas should work in practice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new architecture for continual learning related to sparse distributed memory. It shows that the model works through theoretical analysis and experimental evaluation, relates it to biology and provides good insights.

Some drawbacks (potential future work) include the fact that it does not train the full network but pre-trains convent in a standard way on ImageNet and trains a network that corresponds to one layer MLP on the top of the features. I believe the authors are in the process of extending this but haven't gotten good results yet.
justification_for_why_not_higher_score: It would be good if method worked for full network (future work). 
justification_for_why_not_lower_score: Good results and interesting paper.
Decision: Accept: poster