Sparse Distributed Memory is a Continual Learner
Authors: Trenton Bricken
Xander Davies
Deepak Singh
Dmitry Krotov
Gabriel Kreiman
Venue: ICLR 2023 poster (2023)
Keywords: Sparse Distributed Memory
Sparsity
Top-K Activation
Continual Learning
Biologically Inspired

Abstract: Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner. We find that every component of our MLP variant translated from biology is necessary for continual learning. Our solution is also free from any memory replay or task information, and introduces novel methods to train sparse networks that may be broadly applicable.

Published as a conference paper at ICLR 2023
SPARSE DISTRIBUTED MEMORY IS A CONTINUAL LEARNER
Trenton Bricken
Systems, Synthetic, and Quantitative Biology Harvard University
trentonbrickeneg ha harvard.edd
Xander Davies & Deepak Singh
Computer Science
Harvard College (alexander davies, tejasvisingh) @connnghhhhhllllo @oollege.oleee......oo.mm wew.eeeeaeaaaeee haw.ard.
Dmitry Krotov
Gabriel Kreiman
MIT-IBM Watson Al Lab
Harvard Medical School
IBM Research
Programs in Biophysics and Neuroscience
krotove i bmm. com
K ldrens. .aaansshaavvae harvard.ede edu
ABSTRACT
Continual learning is problem for artificial neural networks that their biologi- cal counterparts are adept solving. Building on work using Sparse Distributed Memory (SDM) to connect core neural circuit with the powerful Transformer model, we create a modified Multi-Laycred Perceptron (MLP) that is a strong continual learner. We find that every component of our MLP variant translated from biology is necessary for continual learning. Our solution is also free from any memory replay task information, and introduces novel methods to train sparse networks that may be broadly applicable.
INTRODUCTION
Biological networks tend thrive in continually learning novel tasks, a problem that remains daunt- ing for artificial neural networks. Here, we use Sparse Distributed Memory (SDM) to modify Multi- Perceptron (MLP) with features from cerebellum-like neural circuit that are shared across organisms as diverse as humans, fruit flies, and electric fish (Modi et al.) 2020; Xie et al.) 2022 These modifications result in a new MLP variant (referred to as SDMLP) that uses a Top- activation function (keeping only the & most excited neurons in layer on), no bias terms, and enforces both normalization and non non-tegatyyy constraints on its weights and data. All these SDM-derived components are necessary for our model to avoid catastrophic forgetting.
We encounter challenges when training the SDMLP that we leverage additional neurobiology to solve, resulting in better continual learning performance. Our first problem is with "dead neurons" that are never active for any input and which are caused by the Top-K activation function (Makhzani & Frey 2014 Ahmad & Scheinkman 2019). Having fewer neurons participating in learning results in more of them being overwritten by any new continual learning task, increasing catastrophic for- getting. Our solution imitates the "GABA Switch" phenomenon where inhibitory interneurons that implement Top-K K will excite rather than inhibit early in development (Gozel & Gerstner 2021).
The second problem is with optimizers that use momentum, which becomes "stale" when training highly sparse networks. This staleness refers to the optimizer continuing to update inactive neurons with an date moving average, killing neurons and again harming continual learning. To our
A called "k Winner Takes All" related literature.

---

Published as a conference paper at ICLR 2023
knowledge, are the first formally identify this problem that will in theory affect any sparse model, including recent Mixtures of Experts (Fedus et al., 2021 Shazeer et al. 2017).
Our SDMLP is strong continual learner, especially when combined with complementary ap- proaches. Using our solution in conjunction with Elastic Weight Consolidation (EWC) (Kirkpatrick) et al. 2017), we obtain, to the best of knowledge, state-of-the-art performance for CIFAR-I0 in the class incremental setting when memory replay is not allowed. Another variant of SDM, developed independently of our work, appears be state-of-the-art for CIFAR-100, MNIST, and FashionMNIST, with our SDMLP as a close second (Shen et al.) [02T].
Excitingly, our continual learning success is "Organic" in resulting from the underlying model archi- tecture and does not require any task labels, task boundaries, or memory replay. Abstractly, SDM learns its subnetworks responsible for continual learning using two core model components. First, the Top-K activation function causes the k neurons most activated by an input to specialize towards this input, resulting the formation of specialized subnetworks. Second, the I,² normalization and absence of bias term together constrain neurons to the data manifold, ensuring that all neurons democratically participate in learning. When these two components are combined, a new learning task only activates and trains small subset of neurons, leaving the rest of the network intact to remember previous tasks without being overwritten
As a roadmap of the paper, we first discuss related work (Section 2b. We then provide a short introduction to SDM (Section 3 before translating it into our MLP (Section 4. Next we present results, comparing the organic continual learning capabilities of our SDMLP against relevant benchmarks (Section 5). Finally, we conclude with a discussion on the limitations of our work, sparse models more broadly, and how SDM relates MLPs to Transformer Attention (Section
2 RELATED WORK
Continual Learning- The techniques developed for continual learning can be broadly divided into
three categories: architectural (Goodfellow et al.] 2014 2013), regularization (Smith et al.
2022
Kirkpatrick et al.
2017; Zenke al. 2017: (Aljundi et al. 2018). and rehearsal (Lange et al.
2021
Hsu et al.] 2018). Many of these approaches have used the formation of sparse subnetworks for
continual learning (Abbasi et al. 2022; Aljundi et al. 2019b Ramasesh et al. 2022: [yer et al.
2022
Xu & Zhu 2018: Mallya & Lazebnik 2018 Schwarz et al.) 2021 Smith et al. 2022 Le
ct al. 2019 Le Venkatesh 2022 1 However, in contrast to our "organic" approach, these methods
employ complex algorithms and additional memory consumption to explicitly protect model weights
important for previous tasks from overwrites F]
Works applying the Top-K activation to continual learning include (Aljundi et al. 2019b Gozel & Gerstner 2021; [yer et al.] (022). SSrivastava et al.] (2013) used a local version of Top-K, defining disjoint subsets of neurons in each layer and applying Top-K locally to each. However, this was only used on simple task-incremental two-split MNIST task and without any of the additional SDM modifications that we found crucial to our strong performance (Table 2).
The Top-K activation function has also been applied more broadly in deep learning (Mkkhzani & Frey 2014 Ahmad & Scheinkman] 2019: SSnguptaa et al., 2018 Gozel & Gerstner 202 Aljundi
ct al. 2019b). Top-K converges with not only the connectivity of many brain regions that utilize inhibitory interneurons but also results showing advantages beyond continual learning, including:
greater interpretability (Makhzani & Frey] [2014) Krotov & Hopfield 2019; Grinberg et al.
2019
robustness to adversarial attacks (Paiton et al.] [2000] Krotov & Hopfield, 2018: [yer et al. 2022
efficient sparse computations (Ahmad & Scheinkman 2019), tiling of the data manifold (Sengupta
al. 2018). and implementation with local Hebbian learning rules (Gozel Gerstner 2021 Sen-
gupta et al. 2018 Krotov & Hopficld 2019 Ryali et al. 2020 Liang et al. (020)).
FlyModel The most closely related method ours is model of the Drosophila Mushroom Body
circuitry (Shen et al., 2021). This model, referred to as "FlyModel", unknowingly implements the SDM algorithm, specifically the Hyperplane variant (Jaeckel 1989a) with Top-K activation
2 'Even without sparsity or the Top-K activation function, pretraining models can still lead to the formation of subnetworks, which translates into better continual learning performance as found in Ramasesh ct al. (2022) Memory replay methods indirectly determine and protect weights by deciding what memories replay.
2

---

Published as a conference paper at ICLR 2023
function that we also use and will justity (Keeler 1988). The FlyModel shows strong continual learning performance, trading off the position of best performer with our SDMLP across tasks and bolstering the title of our paper that SDM is continual learner.
While both models are derived from SDM, our work both extends the theory of SDM by allowing to successfully learn data manifolds (App. A.4), and reconciles the differences between SDM and MLPs, such that SDM can be trained in the deep learning framework (this includes having no fixed neuron weights and using backpropagation), Both of these contributions are novel and may inspire future work beyond continual learning. For example, learning the data manifold preserves similarity in the data and leads to more specialized, interpretable neurons (e.g. Fig. 2f App. H..
Training in the deep learning framework also allows us to combine SDMLP with other gradient- based methods like Elastic Weight Consolidation (EWC) that the FlyModel incompatible with (Kirkpatrick et al., (017). Additionally, demonstrating how MLPs can be implemented as a cere- bellar circuit serves as an example for how ideas from neuroscience, like the GABA switch, can be leveraged to the benefit of deep learning.
BACKGROUND ON SPARSE DISTRIBUTED MEMORY
Sparse Distributed Memory (SDM) is an associative memory model that tries to solve the prob- lem of how patterns (memories) could be stored in the brain (Kanerval 1988 (993) and has with close connections to Hopfield networks, the circuitry of the cerebellum, and Transformer Attention
(Kanerva, 1988 Bricken & Pehlevan 202T Hopfield 1982 1984 Krotov Hopfield 2016 Tyul-
mankov et al. 20 et al.) 2022). We briefly provide background on SDM and notation
sufficient relate SDM MLPs. For summary of how SDM relates to the cerebellum, see App.
A.2 We use the continuous version of SDM, where all neurons and patterns exist on the I,² unit norm hypersphere and cosine similarity our distance metric (Bricken & Pehlevan 2021.
Write patterns into nearby neurons
1
Writeradus
2
Pallemn atored
Writing
Neurons store superposition petterss
neurin
Original pattem location
Neuron
Read patterns from neurons near query
Neuron view:
Query reaci stored patterns. paliem
Graphical Summary of the SDM Write and Read Operations. Top Row Three patterns being written into nearby neurons. 1. First write operation; 2. Patterns are stored inside neurons and the original pattern address is shown; 3. Writing second pattern; Writing third pattern and neurons storing superposition of multiple patterns. Bottom Row The SDM read operation. The query reads from nearby neurons with the inset showing the number of times each pattern is read. Blue is read the most as its original pattern location (shown by the hollow circle) is closest to the query. Fig. adapted with permission from Bricken & Pehlevan (202I).
Figure 1:
SDM randomly initializes the addresses of neurons on the L unit hypersphere in an n dimen- sional space. These neurons have addresses that each occupy column in address matrix X, E (L2)axr, where L shorthand for all n--iienssional vectors existing on the L unit norm hypersphere. Each neuron also has storage vector used to store patterns represented in the matrix
3

---

Published as a conference paper at ICLR 2023
X, E Roxr, where is the output dimension. Patterns also have addresses constrained on the n- dimensional L hypersphere determined by their encoding; encodings can be as simple as flattening an image into vector complex as preprocessing with a deep learning model.
Patterns are stored by activating all nearby neurons within a cosine similarity threshold c, and per- forming elementwise summation with the activated neurons' storage vector. Depending the task at hand, patterns write themselves into the storage vector (e.g.. during a reconstruction task) or write another pattern, possibly of different dimension (e.g., writing in their one hot label for classification task). This write operation and the ensuing read operation are summarized in Fig. D
Because in most cases have fewer neurons than patterns, the by multiple different patterns. This is handled by storing the pattern values in superposition via the aforementioned elementwise summation operation. The fidelity of each pattern stored in this superposition is a function of the vector orthogonality and dimensionality n.
neuron will be activated
Using m to denote the number of patterns, matrix Pa E (L²)uxm for the pattern addresses, and matrix Pe E Roxm for the values patterns want to write, the SDM write operation is:
X₂ P.b(P. X.), b(e) (1, if 2 0, else,
where b(e) performs an element-wise binarization of its input to determine which pattern and neuron addresses are within the cosine similarity threshold e of each other.
Having written patterns into our neurons, we read from the system by inputting a query &, that again activates nearby neurons. Each activated neuron outputs its storage vector and they are all summed elementwise to give final output y. The output can be interpreted as an updated query and optionally L normalized again as a post processing step:
y y= x=x,6(X]5))
(2)
Intuitively, SDM's query will update towards the values of the patterns with the closest addresses. This is because the patterns with the closest addresses will have written their values into more neurons that the query reads from than any competing patterns. For example. in Fig. [ the blue pattern address is the closest the query meaning that it appears the most those nearby neurons the query reads from. SDM is closely related to modern Hopfield networks (Krotov & Hopfield 2016 |amssuer al., 22200] Krotov Hopfield 2220) if they are restricted to single step update of the recurrent dynamics (Tyulmankov et al. 2021 Bricken & Pehlevan 2021 Millidge et al.,
2022).
4 TRANSLATING SDM INTO MLPs FOR CONTINUAL LEARNING
one hidden layer MI.P transforms an input & to an output y. Using notation compatible with SDM and representing unnormalized continuous vectors with tilde, can write the MLP as:
= X.f(E + ba)) + b
(3)
where X. Ruxr and X., Roxr are weight matrices corresponding to our SDM neuron addresses and values, respectively. Meanwhile, b. RR," b, R are the bias parameters and f(-) is the activation function used by the MLP such as ReLU (Glorot et al., 2011).
Using this SDM notation for the MLP, it trivial to see the similarity between the SDM read Eq. 2 and Eq. 3 for single hidden layer MIPs that was first established in (Kanerva 1993). However, the fixed random neuron address SDM makes unable to effectively model real-world data. Using ideas from Keeler 1988), resolve this inability with biologically plausible inhibitory interneu- ron that is approximated by a Top-K activation function. App. A.3 explains how SDM is modified and the actual Top-K activation function used is presented shortly in Eq. 4 This modification makes SDM compatible with an MLP that has no fixed weights. As for the SDM write operation of Eq. this is related to an MLP trained with backprop outlined in App. A.6
The Dead Neuron Problem An issue with the Top-K activation function is that it creates dead
neurons Ahmad & Scheinkman 2019 Rumelhart & Zipser 1985 Makhzani & Frey 2014 Fedus
4

---

Published as a conference paper at ICLR 2023
et al. 2021 Only subset of the randomly initialized neurons will exist closest to the data manifold and be in the top most active, leaving all remaining neurons to never be activated. In the continual learning setting, when new task is introduced, the model has fewer neurons that can learn and must overwrite more that were used for the previous task(s), resulting in catastrophic forgetting.
Rather than pre- ineiiiiiializnn our neuron weights using decomposition of the data manifold (Rumel- hart& & Zipser 1985] McInnes & Healy] 2018) 8) Strang 1993) we ensure that all neurons are active at the start of training so that they update onto the manifold. This approach has been used before but in biologically implausible ways (see App. B.I (Makhzani & Frey (204;) Ahmad & Scheinkman 2019 van den Oord al. D. We instead extend the elegant solution Gozel & Gerstner (2021) by leveraging the inhibitory interneuron we have already introduced. After neurogenesis, when neu- ronal dendrites are randomly initialized, neurons are excited by the inhibitory GABA neurotransmit- ter (see App. (..)). This means that the same inhibitory interneuron used to enforce competition via Top-K inhibition at first creates cooperation by exciting every neuron, allowing them all to learn. As result, every neuron first converges onto the data manifold which point it is inhibited by GABA and Top-K competition begins, forcing each neuron to specialize and tile the data manifold.
We present the full GABA switch implementation in App. B.3 that is the most biologically plausible. However, using the positive weight constraint that makes all activations positive, we found a simpler approximation that gives the same performance. This is by linearly annealing our & value from the maximum number of neurons in the layer down to our desired k. A similar approach without the biological justification is used by Makhzani & Frey (2014) but they zero out all values that not the k most active and keep the remaining k untouched. Motivated by our inhibitory interneuron, instead subtract the 1-th activation from all neurons that remain on. This change leads to a significant boost in continual learning (see Table 2 and App. B.3). Formally, letting the neuron activations pre inhibition be a XT and a" post inhibition:
a' ja- 10-10 11+
descending-sonn sort [a] ((k+1)
h = max (Amargets [kmax - (emmax - /ssrr
where []+ is the ReLU operation, |:J is the floor operator to ensure k; an integer, and descending- sort(.) sorts the neuron activations in descending order to find the k + 1-th largest activation. E, is an integer representing the current training epoch and subscript denotes that K2 and E. change over time. Hyperparameter sets the number of epochs for & to go from its starting value kimax to its target value Kiarget. When k Ainrget our approximated GABA switch is fully inhibitory for every neuron. An algorithm box in App. [...Issmmmrizza how each SDMLP component functions.
The Stale Momentum Problem - We discovered that even using the GABA switch, some optimiz- continue killing off a large fraction of neurons, to the detriment of continual learning. Investigat- ing why, we have coined the "stale momentum" problem where optimizers that utilize some form of momentum (especially Adam and RMSPPrp) fail to compute accurate moving average of pre- vious gradients in the sparse activation setting (Kingma & Ba 2015 Geoffrey Hinton, 2012). Not only will these momentum optimizers update inactive neurons not the Top-K, but also explode gradient magnitudes when neurons become activated after a period of quiescence. We explain and investigate stale momenta further App. and use SGD without momentum as our solution.
Additional Modifications There are five smaller discrepancies (expanded upon in App. A.5 be-
tween SDM and MLPs: (i) using rate codes avoid non differentiable binary activations: (ii) using I normalization of inputs and weights an approximation to contrast encoding and heterosynap- ticity (Sterling & Laughlin, 2015) Rumelhart Zipser 1983 Tononi & Cirelli. (0144;; (irelli]2014);
(iii) removing
bias terms which assume tonic baseline firing rate not present in cerebellar granule cells (Powell) jet al., 2015; Giovannucci et al., 2017); (iv) using only positive (excitatory) weights that respect Dale's s Principle (Dale| 1935); (v) using backpropagation a more efficient (although possibly non- biological (Lillieapp et al. 2020 implementation of Hebbian learning rules (Krotov & Hopfield,
2019 Sengupta al. 2018 Gozel & GGerstner 202
5 SDM AVOIDS CATASTROPHIC FORGETTING
Here we show that the proposed SDMLP architecture results in strong and organic continual learn- ing. In modifying only the model architecture, our approach is highly compatible with other contin-
5

---

Published as a conference paper at ICLR 2023
ual learning strategies such regularization of important weights (Kirkpatrick et al. 2017 Aljundi
ct al. 2018 Zenke et al. 2017) and memory replay (Zhang et al. 2021: Hsu et al. 2018)) both of which the brain is likely to use some fashion. We show that combinations of SDMLP with weight regularization are positive sum and don't explore memory replay.
Experimental Setup- Trying to make the continual learning setting realistic possible, use Split CIFARIO in the class incremental setting with pretraining on ImageNet (Russakovsky al. 2015). This splits CIFARIO into disjoint subsets that each contain two of the classes. For example the first data split contains classes 5 and 2 the second split contains classes 7 and 9, etc. CIFAR more complex than MNIST and captures real- real-world statistical properties of images. The class incremental setting is more difficult than incremental task learning because predictions are made for every CIFAR class instead of just between the two classes in the current task (Hsu et al.) 2018 Farquhar & Gal 2018). Pretraining on ImageNet enables learning general image statistics and is when the GABA switch happens, allowing neurons to specialize and spread across the data manifold. In the main text, present results where our ImageNet32 and CIFAR datasets have been compressed into 256 dimensional latent embeddings taken from the last layer of frozen ConvMixer that was pre-trained ImageNet32 (step #1 of our training regime in Fig. (Trockman & Kolter 2022 Russakovsky et al. 2015
Method
Neurons
k
Val. Acc.
SDMLP
10K
10
0.71
Dataset
Image Embedding
SDMUP
FlyModel
10K
32
0.82
ConvMixer
MAS
10K
NA
0.67
ConvMixer
Net32
SI
10K
NA
0.44
ReLU
10K
NA
0.21
Pre- Train SDMLP
SDMLP
EWC
10K
NA
0.65
Net32
SDMLP+MAS
10K
10
0.84
SDMLP+EWC
10K
10
0.86
Continual Leam SDMLP
Spin FARID
Lear
Oracle
10K NA
0.93
Figure 2: Left, the three-step training regime. Green outlines denote the module currently being trained. Gray hatches on the ConvMixer denote when is frozen. Table 1: Right, Split CIFARIO final validation accuracy. We highlight the best performing SDMLP, baseline, and overall per- former in the 10K neuron setting. Oracle was trained on the full CIFARIÓ dataset. All results are the average of 5 random task splits.
We then reset X,, that learns class labels and switch to continual learning on Split CIFARIO, training every model for 2,000 epochs each of the five data splits (step #3 of Fig. 2. We use such a large number of epuchs to encourage forgetting of previous tasks. We test model performance on both the current data split and every data split seen previously assess forgetting. The CIFAR dataset is split into disjoint sets using five different random seeds to ensure our results are independent of both data split ordering and the classes each split contains.
Using the image embeddings, we pre-train our models ImageNet (step #2 of Fig. D.
We use preprocessed image embeddings because single hidden layer MLPs struggle to learn directly from image pixels. This preprocessing is also representative of the visual processing stream that compresses images used by deeper brain regions (Pisano et al.) 20000: Li et al. [020]] Yamins et al. 2014). However, we consider two ablations that shift the validation accuracies but not the rank ordering of the models or conclusions drawn: (i) Removing the ConvMixer embeddings and instead
training directly on image pixels (removing step #1 of Fig. 2 App.
; (ii) Testing continual
learning without any ImageNet32 pre-training (removing step #2 of Fig. 2 App. F.2 We also run experiments for CIFARI00 (App. F.2). MNIST (App.
F.3 and FashionMNIST (App. F.4
Model Parameters and Baselines All models are MLPs with 1,000 neurons in single hidden layer unless otherwise indicated. When using the Top-K activation function, we set kiunget = 10 and also present Kinget 1. We tested additional & values and suggest how to choose the best Aiaggo value
TTii version of ImageNet has been re-scaled to he 32 by 32 dimensions and is trained for 300 epochs RRusskkvssy 2015).

---

Published as a conference paper at ICLR 2023
in App. C Because the & values considered are highly sparse saving on FLOPs and memory consumption we also evaluate the 10,000 neuron setting which improves SDM and the FlyModel continual learning abilities in particular.
The simplest baselines we implement are the ReLU activation function, L2 regularization, and Dropout, that are all equally organic in not using any task information (Goodfellow et al. 2014). We also compare to the popular regularization based approaches Elastic Weight Consolidation (EWC),
Memory Aware Synapses (MAS), and Synaptic Intelligence (SI) (Kirkpatrick et al.. 2017 Aljundi et al. 2018 Zenke et al. 2017 Smith et al. 2022). These methods infer model weights that are im- portant for each task and penalize updating them by using a regularization term in the loss. To track each parameter, this requires at least doubling memory consumption plus giving task boundaries and requiring a coefficient for the loss term {
In the class incremental learning setting, it has been shown that these regularization methods catas- trophically forget (Hsu et al. [018] Gurbuz & Dovrolis, 2022). However, we found these results to be inaccurate and get better performance through careful hyperparameter tuning. This included adding a 3 coefficient to the softmax outputs of EWC and SI that alleviates the problem of vanishing gradients (see App. G.I.We are unaware of this simple modification being used in prior literature but use it to make our baselines more challenging.
We also compare to related work that emphasizes biological plausibility and sparsity. The FlyModel performs very well and also requires no task information (Shen et al. (02T). Specific training test Active Dendrites SuabilityPPlasticiii Adaptation) (Gurbuz & Dovrolis) 2022). However, these models both use the easier task incremental setting and failed to generalize well to the class incremental setting, especially Active Dendrites, on even the simplest Split MNIST dataset (App. F.3 6 The failure for task incremental methods to apply in a class incremental setting has been documented before (Farquhar& & Gal 2018). These algorithms are also not organic, needing task labels and have significantly more complex, less biologically plausible implementations. We considered a number of additional baselines but found they were all were lacking existing results for the class incremental setting and often lacked publicly available codebases
parameters used for this model can be found in App. G.2 Additionally,
(Iyer et al. 2022) and NISPA (Neuro- Inspird
We present the best continual learning results of each method in Fig. B and its corresponding Table 1. SDMLP organically outperforms all regularization baselines, where its best validation accuracy is 71% compared to 67% for MAS. In the 1K neuron setting (shown in Table 5 of App. F. SDMLP with & = 1 ties with the FlyModel at 70%. In the 10K neuron setting, the FlyModel does much better achieving 82% versus 71% for SDMLP. However, the combination of SDMLP and EWC leads to the strongest performance of 86%. barely forgetting any information compared to an oracle that gets 93% when trained on all of CIFARIO simultaneously
CIFAR1O Results
SDMLP and the regularization based methods are complimentary and operate at different levels of abstraction. SDMLP learns what subset of neurons are important and the regularization method learns what subset of weights are important. Because the FlyModel combines fixed weights with Top-K, it can be viewed as more rigid form of the SDMLP and regularization method com- bined. The FlyModel also benefits from sparse weights making neurons more orthogonal to re- spond uniquely different tasks. We attempted integrate this weight sparsity into our SDMLP so that our neurons could learn the data manifold but be more constrained to a data subset, hypo- thetically producing less cross-talk and forgetting between tasks. Interestingly, initial attempts to prune weights either randomly or just the smallest from the ImageNet pretrained models resulted in catastrophic forgetting. We leave more in depth explorations of weight sparsity to future work.
Ablations We ablate components of our SDMLP to determine which contribute to continual learn- ing in Table 2. Deeper analysis of why each ablation fails is provided in App. H but we highlight
'ee acknowledge that task boundaries can be inferred in an unsupervised fashion but this requires further complexity and we expect the best performance to come from using true task boundaries (Aljundi et al. (019a).
Because we do not test memory replay approaches. we ignore the modified version of NISPA developed for the class incremental setting (Gurhuz & Dovrolis 2022).
7 All of our code and training parameters can be found in our publicly available codebase: https:
git rithub. cttpbo.oomrrreee www./reentrriieeooooooe.om.....e ck
www.erricc/////ooooo. /SStMoontt nua 1 Thea learr
We found that SGDM for the combined SDMLP44WW does slightly hetter than SGD and is used here
while SGD worked the hest for all other approaches.
7

---

Published as a conference paper at ICLR 2023
CIFAR10 Continual Learning Performance
Oracle
SDMLP+EWC
FlyM
SDMLP
MAS
Top-K
ReLU
0.93
0.8
0.5
Validation
0.4
0.2
Task #1
Task #2
Task #3
Task 04
Task #5 Epochs
2k
Figure 3: SDM outperforms regularization methods and their combination is positive sum. We plot validation accuracy across tasks for the best performing methods and baselines on Split CIFARIO. SDMLP+EWC (green line) does the best, then the FlyModel (magenta), then SDMLP (yellow) with MAS (blue) close behind. The Top-K K (orange) without SDM modifications and ReLU only trained for one epoch each task as per Shen al. (202I) but we visually extend the validation accuracy on each task to make method comparison easier. App. F.T visualizes how SDMLP gradually forgets each task compared to catastrophic forgetting of the ReLU baseline. We use the average of 5 random seeds and error bars to show standard error of the mean but the variance small making them hard see.
(purple) baselines do poorly. The FlyModel
the two most important here: First, the Top-K activation funetion prevents more than i neurons from updating for any given input, restricting the number of neurons that can forget information. A small subset of the ncurons will naturally specialize to the new data classes and continue updating towards it, protecting the remainder of the network from being overwritten. This theory is supported by results in App. C.22 where the smaller l is, the less catastrophic forgetting occurs. We provide further support for the importance of k by showing the number of neurons activated when learning each task in App. H
Second, the T.2 normalization constraint and absence of hidden layer bias term crucial to keeping all neurons on the T,2 hypersphere, ensuring they all participate democratically. Without both these constraints, but still using the Top-K activation function, we found that small subset of the neurons become "greedy" in having larger weight norm or bias term and always out-compete the other neurons. This results in a small number of neurons updating for every new task and catastrophically forgetting the previous one as shown in the Table 2 ablations. Evidence of manifold tiling shown Fig. 6 with UMAP (MeInnes & Healy 2018) plot fit on the SDM weights (orange) that were pre-trained on ImageNet32 embeddings. We project the embedded CIFARIO training data (blue) to show that pre-training has given SDM subnetworks that cover the manifold of general image statistics contained in ImageNet32. This manifold tiling is in sharp contrast to the other methods shown in App. H and enables SDM to avoid catastrophic forgetting. As further evidence of manifold tiling, show in the same appendix that SDM weights are often highly interpretable when trained directly on CIFAR10 images.
As additional ablation to SDMLP defined by Eq. in the output layer breaks continual learning. This is because the model assigns large bias values the classes within the current task over any classes in previous tasks, giving the appearance of catastrophic forgetting. Interestingly, this effect only applies to SDM; modifying all of our other baselines by removing the output layer bias term fails affect their continual learning performance.
find that introducing bias term
6 DISCUSSION
Setting out to implement SDM an MLP, introduce number of modifications resulting in model capable of continual learning. These modifications are biologically inspired, incorporating components of cerebellar neurobiology that are collectively necessary for strong continual learning.
8

---

Published as a conference paper at ICLR 2023
Name
Val. Acc.
SGD
0,63
SOM Training Data Fit on Weights
SGDM
0.54
Adam
0.23
RMSProp
0.20
Linear Subtract
0,63
Linear Mask
0.38
No Linear Anneal
0.35
Negative Weights
0.57
No L Norm
0.20
Hidden Layer Bias Term
0.20
12
Output Layer Bias Term
0.20
Fig. 6: SDM learns the data manifold. UMAP representation of CIFARIO embeddings (blue) and SDM weights (orange) pretrained on Ima- geNet32. The overlap between weights and data is lacking in other methods (see App. HI]
Table 2: SDMLP Ablations. Validation accuracy for ablated versions SDMLP on CIFAR10 em- beddings and k = 10 with 1,000 ncurons (the same task as Table 1). See main text for details.
Our new solution to the dead neuron problem and identification of the stale momentum problem key to our results and may further generalize to other sparse neural network architectures.
Being able to write SDM as an MLP with minor modifications is interesting in light of SDM's connection to Transformer Attention (Bricken & Pehlevan 2021). This link converges with work showing that Transformer MLP layers perform associative memory-like like operations that approximate Top-K by showing up to 90% activation sparsity in later layers (Geva et al.) 2020 Sukhbaatar et a
2019 Nelson et al., 2022). Viewing both Attention and MLPs through the lens of SDM presents their tradeoffs: Attention operates on patterns in the model's current receptive field. This increases the fidelity of both the write and read operations because patterns do not need rely on neurons that poorly approximate their original address location and store values in a noisy superposition. However, in the MLP setting, where patterns are stored in and read from neurons, an increase in noise is traded for being able to store patterns beyond the current receptive field. This can allow for learning statistical regularities in the patterns' superpositions to represent useful prototypes, likely benefiting generalization (Irie et al. 2022).
Limitations Our biggest limitation remains ensuring that SDM successfully avoids the dead neuron problem and tiles the data manifold in order to continually learn. Setting kearget and s decay rate remain difficult and data manifold dependent (App. C.2 Avoiding dead neurons currently requires training SDM on static data manifold, whether it is image pixels or fixed embedding. Initial experiments jointly training SDM modules either interleaved throughout a ConvMixer or placed at the end resulted in many dead neurons and failure to continually learn. We believe this is in large part due to the manifold continuing to change over time.
Constraining neurons to exist the data manifold and only allowing a highly sparse subset fire is also suboptimal for maximizing classification accuracy in non continual learning settings. Making sufficiently large can alleviate this problem but at the cost of continual learning abilities. However, the ability for hardware accelerators to leverage sparsity should allow for networks to become wider (increasing r), rather than decreasing k, and help alleviate this issue (Wang 2020 Gale al. 2020). Finally, while our improved version of SDM assigns functionality to five different cell types in the cerebellar circuit, there is more of the circuitry to be mapped such as Basket cells and Deep Cerebellar Nuclei (Sezener et al. 2021).
Conclusion We have shown that SDM, when given the ability to model correlated data manifolds in ways that respect cerebellar neurobiology, is strong continual learner. This result is striking because every component of the SDMLP, including L" normalization, the GABA switch, and mo- mentum free optimization are necessary for these continual learning results. Beyond setting new "organic" baseline for continual learning, we help establish how to train sparse models in the deep learning setting through solutions to the dead neuron and stale momentum problems. More broadly, this work expands the neurobiological mapping of SDM to cerebellar circuitry and relates it to MLP deepening links to the brain and deep learning.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
Thanks to Dr. Beren Millidge, Joe Choo-Choy, Miles Turpin, Blake Bordelon, Stephen Casper, Dr. Mengmi Zhang, Dr. Tomaso Poggio, and Dr. Pentti Kanerva for providing invaluable inspiration, discussions, and feedback. We would also like to thank the open source software contributors that helped make this research possible, including but not limited to: Numpy, Pandas, Scipy, Matplotlib, PyTorch, and Anaconda.
AUTHOR CONTRIBUTIONS
Trenton Bricken conceived of the project and theory, conducted all experiments, and wrote the paper with suggestions from co-authors.
Xander Davies participated in discussions, contributed to early investigations of Dead Neu- rons and Stale Momentum, and reviewed related work.
Deepak Singh reviewed robustness and Top-K related work and participated in discussions.
Dmitry Krotov advised on the continual learning experiments, Top-K learning theory, and relations to associative memory models.
Gabriel Kreiman supervised the project providing guidance throughout on theory and ex- periments.
REFERENCES
Sparse adam. SparseAdam html
Accessed: 2022-05-18.
Ali Abbasi, Parsa Nooralinejad, Vladimir Braverman, Hamed Pirsiavash, and Soheil Kolouri. Spar- sity and heterogeneous dropout for continual learning in the null space of neural activations. ArXiv, abs/2203.06514 2022.
Diego Aguirre. A novel set of weight initialization techniques for deep learning architectures. 2019.
S. Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse representations. ArXiv, 10ts/9933.11277, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11246-11255. 2019a.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. ArXiv, abs/1 806.05421, 2019b.
David Attwell and Simon B. Laughlin. An energy budget for signaling in the grey matter of the brain. Journal of Cerebral Blood Flow & Metabolism, 21:1133 1145, 2001.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Trenton Bricken and Cengiz Pehlevan. Attention approximates sparse distributed memory. NeurIPS, 2021.
Michael Chang. Sid Kaushik, S. Matthew Weinberg, Thomas L. Griffiths, and Sergey Levine. De- centralized reinforcement learning: Global decision decision-making via local economic transactions. In ICML, 2020a.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In ICLR, 2020b.
10

---

Published as a conference paper at ICLR 2023
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Baindoor Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeon- taek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kath- leen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.02311, 2022.
J. A. Connor, H Y Tseng, and P. E. Hockberger. Depolarization- and transmitter-induced changes in intracellular ca2+ of rat cerebellar granule cells explant cultures. In The Journal of neuroscience : the official journal of the Society for Neuroscience, 1987.
Henry Hallet Dale. Pharmacology and nerve-endings (walter emnest dixon memorial lecture); (sec- tion of therapeutics and pharmacology). Proceedings of the Royal Society of Medicine, 28 3: 319-32, 1935.
Peter Dayan and F. Abbott. Theoretical neuroscience: Computational and mathematical modeling of neural systems. 2001.
Matasha Dhar, Adam W. Hantman, and Hiroshi Nishiyama. Developmental pattern and structural factors dendritic survival in cerebellar granule cells in vivo. Scientific Reports, 8, 2018
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. ArXiv, abs/1805.099333, 2018
W. Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, ArXiv, abs/2101.0396 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv: Learning, 2019.
Trevor Gale, Matei A. Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In SC, 2020.
Karunesh Ganguly, Alejandro F. Schinder, Scott Wong. and Mu-Ming Poo. Gaba itself promotes the developmental switch of neuronal gabaergic responses from excitation inhibition. Cell, 105:521-532, 2001.
Kevin Swerksy Geoffrey Hinton, Nitish Srivastava. Neural networks for machine learning lecture 6a overview mini-batch gradient descent. Coursera Deep Learning Lecture Slides, 2012.
Mor Geva, R. Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. ArXiv, Abbs2212..4440 14913, 2020.
Andrea Giovannucci, Aleksandra Badura, Ben Deverett, Farzaneh Najafi, Talmo Pereira, Zhenyu Gao, llker Ozden, Alexander D Kloth, Eftychios A. Pnevmatikakis, Liam Paninski, Chris I. De Zeeuw, Javier Medina, and Samuel S.-H. Wang. Cerebellar granule cells acquire a widespread predictive feedback signal during motor learning. Nature Neuroscience, 20:727-734, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011.
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, and Yoshua Bengio. Maxout networks. In ICML, 2013.
11

---

Published as a conference paper at ICLR 2023
lan J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient- based neural networks. CoRR, abs/1312.6211,, 2014.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. Nature, 521:436-444, 2015.
Olivia Gozel and Wulfram Gerstner A functional model of adult dentate gyrus neurogenesis, eliff,, 10, 2021.
Lcopold Grinberg, John J. Hopfield, and Dmitry Krotov. Local unsupervised learning for image analysis. ArXiv, abs/1908 08993, 2019.
Charles G. Gross. Genealogy of the "grandmother cell", The Neuroscientist, 8:512-515 2002.
Mustafa Burak Gurbuz and Constantine Dovrolis. Nispa: Neuro-inspired stability-plasticity adapta- tion for continual learning in sparse networks. ArXiv, abs/2206.09117, 2022.
Stefanie Heigele, Sébastien Sultan, Nicolas Toni, and Josef Bischofberger. Bidirectional gabaergic control of action potential firing in newborn hippocampal granule cells. Nature Neuroscience, 19: 263-270, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: Learning. 2016.
JJ Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 796))2554--2588 1982. ISSN 0027-8424. doi: 0ttp:3//paa.7....25555. URL https: www pnas wtt/ ottg///oteet//79///2222
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10):0088-399 1984.
Eriola Hoxha, F. Tempia, P. Lippiello, and M. C. Miniaci. Modulation, plasticity and pathophysiol- ogy of the parallel fiber-purkinje cell synapse. Frontiers in Synaptic Neuroscience, 8, 2016.
Yen-Chang Hsu, Yen-Cheng Liu, and Zsolt Kira. Re-evaluating continual learning scenarios: A
categorization and case for strong baselines. ArXiv, abs/1810.12485, 2018.
Al Impacts. Neuron firing rates in humans, May 2022. rate-off-neuron-fl. firinn
URL https://aiimpacte
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. ArXiv, abs/1502.03167, 2015.
Kazuki Irie, R' o'obert Csord' as, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting time predictions training patterns via spotlights of attention. In ICML, 2022.
Abhiram Iyer. Karan Grewal, Akash Velu, Lucas O. Souza, Jeremy Forest, and Subutai Ahmad. Avoiding catastrophe: Active dendrites enable multi-task learning in dynamic environments. In Frontiers Neurorobotics, 2022.
Louis A. Jaeckel. A class designs for sparse distributed memory. 1989a.
Louis A. Jaeckel. An alternative design for sparse distributed memory. 1989b.
P. Kanerva. Sparse distributed memory and related models. 1993.
Pentti Kanerva. Sparse Distributed Memory. MIT Pr., 1988.
James D. Keeler. Comparison between kanerva's sdm and hopfield-type neural networks.
Cognitive Science, 12(3):299 329, 1988. ISSN 0364-0213. doi: https://doi.rr//
10.101660366-02218889900-------0 URL http://www....eecceeeceeeccaaaaa axticle//S//0380201888888
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,
abs/1412.699s 2015.
12

---

Published as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agmieszka Baroskkka,r,iskkuu, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2017.
D. Krotov and J. Hopfield. Large associative memory problem neurobiology and machine learn- ing. ArXiv, abs/2008. 06690, 2020.
Dmitry Krotov and John Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Dmitry Krotov and John J. Hopfield. Dense associative memory is robust to adversarial inputs. Neural Computation, 30:3151-3167, 2018.
Dmitry Krotov and John J. Hopfield. Unsupervised learning by competing hidden units. Proceedings of the National Academy of Sciences of the United States of America, 116:7723 7731, 2019.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ale Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, PP, 2021.
Frederic Lanore, N. Alex Cayco-Gajic, Harsha Gurnani, Diccon Coyle, and Robin Angus Silver. Cerebellar granule cell axons support high dimensional representations. Nature neuroscience, 24: 1142 1150, 2021.
Hung Le and Svetha Venkatesh. Neurocoder: General-purpose computation using stored neural programs. In International Conference on Machine Learning, 2022.
Hung Le, T. Tran, and Svetha Venkatesh. Neural stored-program memory. ArXiv, abs/1906.08862, 2019.
Jinzhi Li, Brenna Mahoney, Miles Solomon Jacob, and Sophie Jeanne Cécile Caron. Visual input into the drosophila melanogaster mushroom body. bioRxiv, 2020.
Yuchen Liang Chaitanya Ryali, Benjamin Hoover, Leopold Grinberg, Saket Navlakha, Mo- hammed Zaki, and Dmitry Krotov. Can fruit fly learn word embeddings? In International Conference on Learning Representations, 2020.
Timothy P. Lillicrap, A. Santoro, Luke Marris, C. Akerman, and G. Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21:335-346, 2020.
Andrew C. Lin, Alexei M. Bygrave, Alix de Calignon, Tzumin Lee, and Gero Miesenböck. Sparse, decorrelated odor coding the mushroom body enhances learned odor diserimination. Nature neuroscience, 17:559 568, 2014.
Ashok Litwin- Kumar, D. Harris, Axel, H. Sompolinsky, and Abbott. Optimal degrees of synaptic connectivity. Neuron, 93:1 15:1153-11644 164.e7, 2017.
Alireza Makhzani and Brendan Frey. k-sparse autoencoders. CoRR, abs/1312.5663, 2014.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks a single network by iterative pruning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765- 7773, 2018
D. Marr. A theory of cerebellar cortex. The Journal of Physiology, 202, 1969.
Leland McInnes and John Healy. Umap: Uniform manifold approximation and projection for di- mension reduction. ArXiv, abs/1802.03426,, 2018.
Beren Millidge, Tommaso Salvatori. Yuhang Song, Thomas Lukasiewicz, and Rafat Bogacz. Uni- versal hopfield networks: A general framework for single-shot associative memory models. ArXiv, 0ns22202.04557. 2022.
13

---

Published as a conference paper at ICLR 2023
M. Modi, Yichun Shuai, and G. Turne. The drosophila mushroom body: From architecture to algorithm in a learning circuit. Annual review of neuroscience, 2020.
Elhage Nelson, Hume Tristan, Olsson Catherine, Nanda Neel, Henighan Tom, Johnston Scott, ElShowk Sheer, Joseph Nicholas, DasSarma Nova, Mann Ben, Hemnandez Danny, Askell Amanda, Ndousse Kamal, Jones, Drain Dawn, Chen Anna, Bai Yuntao, Ganguli Deep, Lovitt Liane, Haaffied--Dddss Zac, Kernion Jackson, Conerly Tom, Kravec Shauna, Fort Stanislav, Kada- vath Saurav, Jacobson Josh, Tran-Johnson Eli, Kaplan Jared, Clark Jack, Brown Tom, McCandlish Sam, Amodei Dario, and Olah Christopher. Softmax linear units. Transformer Circuits Thread, 2022. http:///oooooooroooooooo////////
Dylan M. Paiton, Charles G Frye. Sheng Y. Lundquist, Joel Bowen, Ryan Zarcone, and Bruno A. Olshausen. Selectivity and robustness of sparse coding networks. Journal of Vision, 20, 2020.
Dabal Pedamonti. Comparison of non-linear activation functions for deep neural networks mnist classification task. ArXiv, abs/1 Abs//804.ng 004402263, 2018.
Thomas John Pisano. Zahra M. Dhanerawala, Mikhail Kislin, Dariya Bakshinskaya, Esteban A. En- gel, Ethan J. Hansen, Austin Hoag, Junuk Lee, Nina L. Oude, Kannan Umadevi Venkataraju, Jessica L.. Verpeut, Freek E. Hoebeek, Ben D. Richardson, Henk-Jan Boele, and Samuel S.-H. Wang. Homologous organization of cerebellar pathways to sensory, motor, and associative fore- brain. bioRxiv, 2020.
Kate Powell, Alexandre Mathy, Ian Duguid, and Michael Häusser. Synaptic representation of loco- motion in single cerebellar granule cells. eLife, 4, 2015.
D Purves, GJ Augustine, and D Fitzpatrick. Neurons Often Release More Than One Transmitter. Sinauer Associates, 2nd edition, 2001.
Rodrigo Quian Quiroga, Leila Reddy, G. Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 435:1102-1109 2005.
Vinay V. Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of model and pretraining scale on catastrophic forgetting in neural networks. 2022.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gru- ber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is you need, 2020.
Stephen Roller, Sainbayar Sukhbaatar, Arthur D. Szlam, and Jason Weston. Hash layers for large sparse models. In NeurlPS, 2021.
David E. Rumelhart and David Zipser. Feature discovery by competitive learning. 1985.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
Chaitanya Ryali, John Hopfield, Leopold Grinberg, and Dmitry Krotov. Bio-inspired hashing for unsupervised similarity search. In International Conference on Machine Learning, pp. 8295- 8306. PMLR, 2020.
Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter E. Latham, and Yee Whye Tch. Powerpropagation: A sparsity inducing weight reparameterisation. In NeurIPS, 2021.
Anirvan M. Sengupta, M. Tepper, C. Pehlevan, A. Genkin, and D. Chklovskii. Manifold-tile tiling localized receptive fields are optimal in similarity-preserving preserving neural networks. bioRxiv, 2018.
Biswa Sengupta, Martin B Stemmler, Simon B. Laughlin, and Jeremy E. Niven. Action potential en- ergy efficiency varies among neuron types in vertebrates and invertebrates. PLoS Computational Biology, 6, 2010.
14

---

Published as a conference paper at ICLR 2023
Eren Sezener, Agnieszka Crabska-Barwinskar Dimitar Kostadinov, Maxime Beau, Sanjukta Krish- nagopal, David Budden, Marcus Hutter, Joel Veness, Matthew M. Botvinick, Claudia Clopath, Michael Häusser, and Peter E. Latham. A rapid and efficient learning rule for biological neural circuits. bioRxiv, 2021.
Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. ArXiv, abs/1701.065334 2017.
Yang Shen, Sanjoy Dasgupta, and Saket Navlakha. Algorithmic insights on continual learning from fruit flies. ArXiv, abs/2107.07617, 2021.
James Smith, Junjiao Tian, Yen-Cham Hsu, and Zsolt Kira. A closer look at rehearsal -free continual learning. ArXiv, alts/223..126.. 2022.
Rupesh Kumar Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmidhuber. Compete compute. In NIPS, 2013.
Peter Sterling and Simon Laughlin. Principles of neural design. 2015.
Gilbert Strang, Introduction to linear algebra. 1993.
Sainbayar Sukhbaatar, E. Grave, Guillaume Lample, H. Jégou, and Armand Joulin. Augmenting self-attention with persistent memory. ArXiv, abs/1907.01479 00040, 2019.
Giulio Tononi and Chiara Cirelli. Sleep and the price of plasticity: From synaptic and cellular homeostasis to memory consolidation and integration. Neuron, 81:12-34, 2014.
Asher Trockman and J. Zico Kolter. Patches are all you need? ArXiv, hhs/2201.099992 2022.
Danil Tyulmankov, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang. Biological key- value memory networks. Advances in Neural Information Processing Systems, 34, 2021.
Aäron van den Oord, Oriol Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In NIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Ziheng Wang. Sparsert: Accelerating unstructured sparsity on gpus for deep learning inference. Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques, 2020.
Marjorie Xie, Samuel P. Muscinelli, Kameron Decker Harris, and Ashok Litwin-Kumar Task- dependent optimal representations for cerebellar learning, bioRxiv, 2022.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurIPS, 2018.
Daniel Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimizzed hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111:8619 - 8624, 2014.
Chris I. De Zeeuw. Bidirectional learning in upbound and downbound microzones of the cerebellum. Nature Reviews Neuroscience, 22:92-110, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987-3995. 2017.
Mengmi Zhang, Rohil Badkundri, Morgan B. Talbot, and Gabriel Kreiman. Hypothesis-driven stream learning with augmented memory. ArXiv, abs/2104.02206, 2021.
15

---

Published as a conference paper at ICLR 2023
App.
Table of Contents
SDM
16
A.1
SDM Training Algorithm
16
A.2
SDM Biological Plausibility
17
Learning SDM Neuron Addresses with the Top- K Activation Function
20
A.4
Why SDM Originally Required Fixed Neuron Addresses
20
Additional Modifications to SDM
21
A.6
SDM Write Operation Relation MLP Backpropagation
23
A.
Rate Code Activations Maintain SDM's Approximation to Transtormer Atlention
23
B
GABA Switch
26
B.1
Biologically Implausible Solutions the Dead Neuron Problem
26
B.2
GABA Switch Biology
26
B.3
GABA Switch Practical Considerations
28
C
Top-K
31
C.I
Inhibitory Internurrons Approximate Top-K
31
Optimized Top-K
D Stale Momentum
36
E Training Regime Ablations
40
E.1 Training Directly Image Pixels
40
No Pretraining
F
Additional Datasets
43
F1
CIFARTO Extra Fig. and Table
43
CIFARTOD
43
F3
Split MNIST
46
F.4
Split PashionMNIST
47
G
Baseline Implementations
48
G.
Beta Coefficient for Elastic Weight Consolidation and Synaptic Intelligence
48
2
FlyModel Parameters
49
Investigating Differences in Continual Learning Abilities
49
A
SDM
A.1 SDM TRAINING ALGORITHIM
Here present in full the training loop used for the SDMLPP
16

---

Published as a conference paper at ICLR 2023
Algorithm SDMLP Training Algorithm
Input: Model weights: Xy), Tasks: (Ti,..., Ta), Top-K params: kmax ktarget. Train steps: X,), Output: Trained model weights: (Xa;Xp) X, + LayerInit(X, (X,,
// Reset neuron output weights // Obtain data and class labels for curzent task // Sample a data point from the task // L2 normalize the data // Get neuron activations // Apply Top-K Eq.4 // Get model predictions
for i in (I..... .11 do
X. Y < T;
for in (I, (l)...,S) do
x.y ~ X,
x/||x|
a f --X,,x
a + Top-K(a)
ý + X a
VaXa, VaX, t ComputeGradients(/
// Compute Loss and Gradients // Update model weights
Xay X, + GradientStep( VaXa, VoX.)
Xa. X2 f- [Xa]+, [X,1+
// clamp all weights to be positive // L2 normalize the neuron addresses
Xa t Xa |aal
end end
return Xa X,
A.2 SDM BIOLOGICAL PLAUSIBILITY
Here we summarize the biological foundations of SDM 9 The biological plausibility of our new modifications to SDM that allow it learn the data manifold and be implemented as deep learning model can be found in App. A.5 The biological plausibility of the GABA switch can be found in App. B.2
Fig. 5 overlays SDM notation and operations on the cerebellar circuitry. Mossy Fibers represent in- coming queries & and pattern addresses Pa through their firing activity. Each Granule cell represents an SDM neuron x, with its dendrites the neuron address vector x., and post-synaptic connections with Purkinje cells as the storage vector x,, branching perpendicularly, each Granule cells axon (called a Parallel Fiber), forms contacts with thousands of Purkinje cells with many granule cells (Hoxha et al. 2016). The strength of each contact represents a counter recording the value a particular element in the neuron storage vector (x,)i. where i indexes this vector element.
During SDM read operations, this Purkinje cell can thus sum over index all activated neuron storage vectors and use its firing rate to represent the value of y;. During SDM write operations, the Climbing Fibers are used to update the Purkinje cell counter of each active neuron with the pattern value at this index position (pr):. The most demanding part of SDM's implementation in a biological circuit that is strikingly satisfied by the cerebellum is the three way interface and specificity that exists between Purkinje cells, Climbing Fibers, and Parallel Fibers. This allows for Granule cells to have storage vectors that are precisely written to and read from. We also provide Fig. 6 to give an additional perspective on the biological mapping.
While our improved version of SDM assigns functionality to five different cell types in the cerebel- lar circuit, there is more of the circuitry be mapped such the Basket cells and Deep Cerebellar Nuclei (Sezener et al. 2021). There is also additional neuroscientific evidence required to confirm that cerebellum operates as SDM predicts, for example that granule cells fire sparsely (Kan- erva 1988 Bricken & Pehlevan, 2021 Giovannucci et al.) 2017). Within the scope of our model, the weakest biological plausibility is how well Top- K actually approximates the Golgi inhibitory interneuron (App. C.I and how L normalization is implemented (App. A.5.
'A more extensive treatment of how SDM may he implemented by the cerchellum can be found in Chapter 9 of the SDM book (Kanerval 1988 and (Kanerva 1993).
17

---

Published as a conference paper at ICLR 2023
St
X
#5 [X
Ba
Pa
Go
#3
X
X
Gr
#6
2
#4 4
#1 A
Mo
CI
Pa
[Pvl [Y]
Figure 5: SDM Mapping the Cerebellum. The operations are as follows: #1. A pattern Pa (blue) or query & (red) enters via firing of the Mossy Fibers (Mo). #2. Mossy fibers activate Granule Cells (Gr), with the Gr dendrites representing X., Here we show two different Granule cells and their addresses x3 and x². These Granule Cells project their axons (called Parallel Fibers (Pa)) upwards and across Purkinje Cells (Pu) where their synaptic connections encode their storage vectors. Each contact with Purkinje Cell encodes single element of the storage vector, Here show of these elements with the first Purkinje Cell with the subscript 1, [x]]1 and [x]]1- We will come back these. #3. A Golgi (Go) inhbitory interneuron gets input from number of neuron types including Granule cells, it inhibits these Granule cells with a value of - 7, implementing an approximate Top-K. #4. If and only if we are writing a pattern P., comes through climbing fiber (CI) that wraps around a single Purkinje Cell to produce large action potentials that induce long term potentiation/ddeprsssion (LTP/LTD). Here we show the Climbing fiber that writes to the Purkinje Cell representing the first element of the storage vectors with [Pol1- #5. The Granule Cells that are firing excite the Purkinje cells in proportion represent their storage vectors. In the case of writing, this updates these synapses. #6. If we are reading from the system, the Purkinje Cell integrates signals from activate neurons" storage vectors and determines whether not fire, implementing non-linear activation function that outputs [y] (the first element the vector in this case). The neuron types not mentioned previously are, going clockwise starting in the top right: St=Stellate cell (inhibitory interneuron) and Ba=Basket cell (inhibitory interneuron). Displayed originally without the SDM notation overlayed as Fig. 3.11 of (Kanerva] 1993), reprinted with permission from Pentti Kanerva whom all rights belong.
the strength of their synaptic connections that
18

---

Published as a conference paper at ICLR 2023
Return Majority Bit
Purkinje Cellin
Countern
Write Pattern B
Climbing Fibern
Majority
Purkinje Cell
Counter
Pattern
Climbing Fiber
Return Majorit Bit
Purkinje Cell 1
Counter
Counter
White Patter Bit 1
Climbing Fiber
Address Decoder
Granute
Dendrites
Read/Write Address Input
Moszy Fibers
Neuron
Granule
Granule
(b)
Figure 6: Another perspective the mapping from SDM to the cerebellar circuitry. (a) The computational circuitry necessary to implement SDM. (b) The biological implementation of SDM where the labels of (a) have been replaced by their biological components. The Purkinje cell is shown as an oval to represent the synaptic contacts with the granule cells. See the text for explana- tions of the architectural features.
19

---

Published as a conference paper at ICLR 2023
A.3 LEARNING SDM NEURON ADDRESSES WITH THE TOP-K ACTIVATION FUNCTION
SDM was built on the assumption that its neuron addresses, Xa, are randomly initialized and fixed in place (Kanerva 1988). This was for reasons of both biological plausibility and analytical tractability when determining the maximum memory capacity and convergence dynamics of the model (see App. A.4 However, these random and fixed neuron addresses will often not be within cosine similarity of the real-world patterns existing on some lower dimensional manifold in the vector space. As result, these neurons will often never be written to or read from.
Keeler 1988) outlined how SDM could both retain its theoretical results and allow for neuron ad- dresses to learn the data manifold. This is by replacing the fixed cosine similarity activation thresh- old c that activates variable number of neurons, with a variable threshold that activates fixed k neurons. Intuitively, with random neuron and pattern addresses, a fixed c would result in a pattern or query activating some k neurons in expectation, keeping neuron utilization constant. However, using the same fixed e with non-random addresses would vary the number of neurons being acti- vated and utilized. Over-utilized neurons would store too many pattern values in their storage vector superposition, harming pattern fidelity. Instead, we can achieve the same constant neuron utilization as in the random address setting by varying c such that only k neurons are always activated.
How can e be dynamically adjusted to keep a constant k neurons active in a biologically plausible fashion? Via inhibitory interneuron that creates negative feedback loop: the more neurons that are active, the more activated the interneuron becomes, and the more it inhibits, keeping only k active at convergence 10 Sparsity inducing interneurons are ubiquitious across layers of cortical processing in the brain, particularly relevant to SDM is the cerebellar Golgi interneuron (Marr 1969. Keeler 1988 Paiton et al. 2020).
A.4 WHY SDM ORIGINALLY REQUIRED FIXED NEURON ADDRESSES
In attempting to be biologically plausible, SDM was designed to respect Dale's Principle, whereby a synaptic connection can be either excitatory or inhibitory but not transition between them (Dale) [935). Using the original binary vector formulation, neurons could compute they were near enough to incoming pattern query to be activated (within some Hamming distance threshold) through the following steps:
1. Converting the neuron's binary address to bipolar weights {+1, corresponding to be- ing excitatory or inhibitory, respectively.
2. Activating those weights where the incoming query has 1 in its address and summing over them. This corresponds to how many Is the query and neuron address vectors agree - if the addresses match then excitatory weight is activated, giving +1, if they disagree then a negative weight is activated, giving -1.
3. Rescaling the binary Hamming distance threshold if the neuron is activated by the total number of is in the neuron address. See (Kanerva 1988 1993) for details beyond this summary.
Formally, the binary neuron address can be converted into weights w. = +1 with:
W/ (2 +x) 1, E [1,..., n).
(5)
Where 7 is used to denote a specific neuron address, the query vector is &, and the function to determine if the neuron should be activated is al:; .) that returns binary action potential. We can write:
[
a(E, -1
[[_ wiS
> d
Σ
(2=
(2 (xx) (xxxx)) 1))
> d
where there is the action potential threshold d for the neuron to fire and [[-] is the indica- tor function. The interval of values for address decoding is the number of Is in the address,
--- ] [x]]] By adjusting our value account for the possible number of matches, we can implement the Hamming distance threshold for each neuron.
App- C.1 discusses of how well inhibitory interneurons be approximated by Top-K.
20

---

Published as a conference paper at ICLR 2023
The potential for changing the magnitude of neuron weights was considered but change of their sign was banned due to Dale's Principle. It was acknowledged in (Kanerval 1988) that changing the magnitudes of the neuron weights results in a weighted sum where the matching of some bits in the addresses matters more than others for calculating if the neuron is activated. However, the crucial feature is the sign of the weights, determining if there is match mismatch. This sign change is what was banned by Dale's Principle, making the neuron addresses fixed both to their sign and to their specific value by the use of binary vectors that cannot represent nuances in weight magnitude (Kanrva
988 1993).
However, the original SDM approach still fact violated Dale's Principle by mapping the binary neuron address to bipolar because of what this means for the pre-synaptic input neurons. Consider the 3rd element of the input vector (either a pattern or query) being on (a value). The neuron that is active and represents this value will not have a mixture of both excitatory and inhibitory efferent connections with the SDM neuron addresses. In the cerebellar mapping, mossy fibers represent the pattern/query inputs and release glutamate. This means they are always excitatory to granule cells and the Os a neuron address should correspond to having no weight (no dendritic connection) rather than negative weight (inhibitory connection).
An additional issue with the original SDM formulation that the neuron addresses are dense vectors; this is not the case for the granule cells that they mapped onto in the cerebellum where each has only ~ dendrites (Litwin- (Litwin-Kumar et al. 2017) (see (Jaeckel 1989a) for a solution that makes the weights of SDM sparse).
Taken together, these issues can be resolved by having the binary neuron addresses be sparse and staying binary rather than becoming bipolar. We can then allow our neuron addresses use positive real values, simultaneously allowing for changes in weight strengths and respecting Dale's Principle. In this work, we use positive real values but do allow for our weights to be dense, leaving high degrees of weight sparsity to future exploration (Jacckel 1989a
A.5 ADDITIONAL MODIFICATIONS TO SDM
The five modifications made to SDM for it to be implemented in a deep learning framework and that differentiate it from vanilla MLP are explained in full here. These modifications are:
1. Using continuous instead of binary neuron activations
2. 1,2 normalization of inputs and weights
3. No bias term
4. Only positive weights
5. Backpropagation
1. Continuous Activations SDM originally modelled neurons as having binary action potentials by using a Heaviside step function. However, this is non-differentiable and we want to use backprop- agation for training our model. This can be resolved by viewing a neuron's 's action potentials over time interval, referred to in the neuroscience literature as a rate code (Dayan & Abbott 2001). We believe this is compatible with SDM, whereby neurons with addresses closer to an incoming pattern or query will receive stronger stimulation and fire more action potentials by having more dendritic connections stimulated by excitatory neurotransmitters.
We can represent neuronal activation as an expected firing rate with real positive number E [0, f] where t some maximum firing rate. This may look like a rectified tanh function where, due to refractory periods, there are diminishing returns to more stimulation (Glorot et al.) (011). However, because our neurons are already constrained by T,2 normalization to be in the cosine similarity bounds of [-1, 1] we simply use ReLU activation function.
Implementing weighted read and write operations in our original SDM Eqs. [ and we would replace our binarizing function b(-) with a weight coefficient proportional to the distance between the input and neuron addresses. We show in App. A.7 that this modification has minimal impact on how SDM weights different patterns. This means it should have minimal effect memory capacity and is still approximately exponential, maintaining its relationship to Transformer Attention (Bricken & |Pehlevan 2021).
21

---

Published as a conference paper at ICLR 2023
2. L Normalization SDM requires a valid distance metric to compute if neurons and pat- terns/queries are sufficiently close to each other to produce an activation. We make SDM continuous so that it is differentiable by following (Bricken & Pehlevan 202I) and replacing its original Ham- ming distance with cosine similarity.
In SDM's mapping the cerehellum (outlined in App. A.2 mossy fibers represent input pat- terns/queries and granule cell dendrites represent neuron addresses. remains to be experimentally established if and how these specific cell types enforce L normalization. However, contrast en- coding and heterosynapticity are both ubiquitous mechanisms that can approximate L normaliza- tion for the mossy fiber activations and granule dendrites, respectively (Sterling & Laughlim 2015] Rumelhart& & Zipser [985] Tononi & Cirelli] 2014). From a deep learning perspective, LayerNorm and BatchNorm are both used ubiquitously and can also be viewed as approximations to 1,² normal- ization (Toffe& & Szegedy 2015
Ba et al. 2016).
3. No Bias Term- In order to have our neuron activations represent cosine similarities, we must remove the bias term. We also remove the bias term from the output layer so that outputs represent only a summation of the neuron storage vectors that are activated. The absence of both of these bias terms is clear in the SDM Eq. 2 compared to the MLP Eq. 3
Bias terms can be viewed biologically as representing a neuron activation threshold (if negative) or a baseline tonic firing rate (if positive). In SDM's cerebellar mapping, granule cells that represent the neurons do not have a tonic bascline firing rate meaning that positive bias terms should not be allowed (Powell et al., 2015 Giovannucci et al., 2017). Purkinje cell firing that represents the output layer is much more complex such that keeping or removing the bias term is hard to justify biologically (Zeeuw 2020 but we follow the SDM equation in also removing it.
While removing positive bias terms from the granule cells fits neurobiology, removing negative bias terms corresponding an activation threshold is less justified. However, it relates learning dynamics and the ability for Top-K to still be approximated, the ordinary differential equations of (Gozel& & Gerstner 2021) (summarized in [.I). still maintain approximately k neurons firing while using activation thresholds. This is because the fewer neurons that are firing, the less the inhibitory interneuron is activated, keeping more neurons active. As result, and in line with SDM Eq. 2 we make the simplifying assumption of not allowing for negative bias terms either, thus removing bias terms entirely.
An additional benefit of SDM having no bias term, in conjunction with positive weights, is that all neurons will have positive activations by default. This allows for i annealing instead of the GABA switch that is otherwise needed to inject positive current into each neuron when learning the data manifold (also discussed in App. B.3). Having all neurons active also guarantees that there will always be at least i active neurons to use in the Top-K.
is noteworthy that the removal of the bias terms is seeing a resurgence in state of the art models such as the 540B parameter PaLM Transformer language model (Chowdhery et al. 2022). which noted that removal of bias terms resulted in more stable training.
4. Positive Weights The connection between mossy fibers and granule cells is excitatory so the weights should be positive. Allowing for both positive and negative weights would violate Dale's Principle (Dale 1935). As discussed in the last section, the combination of only positive weights and no bias term gives the added benefit of ensuring that all neuronal activations are positive, allowing for the use of the simpler k annealing algorithm.
Coincidentally, having only positive weights also gives better continual learning performance as shown in the ablations of Table 2. A final benefit the creation of sparse weights but this advantage has its limitations. In models trained on our ConvMixer embeddings, we found only ~20% weight sparsity. Meanwhile, when trying to jointly train ConvMixer with SDM on top, too many weights were set to 0, resulting in failed training runs.
5. Backpropagation Gradient descent via backpropagation used to train MLPs is likely to be biologically implausible (Lillicrap et al., 2020); Goodfellow et al.] 2015). However, there exist number of local Hebbian learning rules associated with inhibitory interneurons and manifold tiling that we viable alternatives (Sengupta et al. 2018 Krotov & Hopficld 2019 Gozel & Gerstner
22

---

Published as a conference paper at ICLR 2023
2021). In addition, the cerebellum receives supervised learning signals, via climbing fibers that are known to be capable of encoding heteroassociative relationships such as an eyeblink response to tone. This circuitry is considered in (Sezener et al. 2021) and presents yet another alternative to backpropagation for training the cerehellum.
A.6 SDM WRITE OPERATION RELATION MLP BACKPROPAGATION
The original SDM write operation directly updates the neuron value vector with the pattern value X, = X, op, where weights the pattern by the amount each neuron was activated. Meanwhile, the backpropgation write operation updates true class one-hot (using cross entropy loss). During model training all inputs are considered write operations. After training, during inference, all inputs are considered queries that perform the SDM read operation. Pa corresponds to CIFAR image and Pa one hot label of its encoding.
with the eπror between the model output and the
There are few reasons why this difference is compatible with SDM:
First, an optimal solution for Xy that will result in zero error the one hot encoding used by the original SDM write operation.
Second, the original SDM write is only appropriate when the neuron addresses are fixed and x, is the only thing learnt. Otherwise, neurons update their address X,, this changes the patterns they are activated by and what P. they should store in X,. Using the backpropagation approach to continuously update x,, as function of the patterns it is currently activated by is a viable solution.
Finally, from a biological perspective, the error signal used by backpropagation is closer approxi- mation to how the cerebellar circuit that SDM maps to updates x,, (Bidirectional learning in upbound and downbound microzones of the cerebellum, De Zeeuw, 2021). While backpropagation through multiple layers of deep network has been argued be biologically implausible, this update the output layer is directly connected to the error computation making possible.
In summary, SDM will explicitly write in Pa while the MLP with backprop will compute delta between p. and the network output but this approach is compatible with the same solution, works better when also learning neuron addresses, and is likely to be more biologically plausible.
A.7 RATE CODE ACTIVATIONS MAINTAIN SDM'S APPROXIMATION TO TRANSFORMER ATTENTION
It was shown in (Bricken & Pehlevan 2021 that the Attention operation of Transformers closely approximates the SDM read operation Vaswani al. 2017). This because the weighting assigned to each pattern in SDM is approximately exponential, resulting in the softmax rule.
However, this result is derived where SDM has binary activations of neurons when reading and writing, not weighting each pattern by its distance. Here, we show that with linear or exponential pattern weightings, SDM remains a close approximation to Attention. This makes the results of previous work continue to hold in case where SDM is written as MLP, giving interesting relation between MLPs and Attention (see Section 6).
The algorithm derived in (Bricken & Pehlevan 2021) for the size of each circle intersection is summarized before showing how it can implement the weighting coefficient and the effects this weighting.
As in the original SDM formulation, we are using n dimensional binary vectors with a Hamming activation threshold of d and the distance between the addresses of a query & and pattern pa is d, == d(E. P.). We can group the elements of & and pa into two disjoint sets: the n - d, elements
It is an open question if the climbing fiber signals encode emrors or the target he learnt but "supervised" here as superset of the two and distinet from unsupervised learning without any "teacher" signal.
23

---

Published as a conference paper at ICLR 2023
where they agree and the d, elements where they disagred
n-d,
0.
0.
Pa
0,
a+2
b+c
Now imagine third vector, representing a potential neuron address. This neuron has four possible groups that the elements of its vector can fall into:
agree with both & and Pa
disagree with both & and Pa
agree with & and disagree with Pa
c agree with Pa and disagree with E
We want to constraint the values of 2, and such that the neuron address exists inside the circle intersection between and Рa. This produces the following constraints:
a + b + c + = n
+ b 2 d
a atc c 2 n d dy
4 b b+c c dp
Using the notation of (Vaswani et al. 2017), we can write the total number of neurons that exist in the intersection the read and write circles as:
dy (o-(n-d-p
WType di ((aaa)
(c.) )
4=
d-a)
where introduce the weight coefficient wType that can be: WBinnry (a, de, n) =
WLinear (a, du n) + a ado c)
n
n
WExp (a, c, de, n) exp - 3(n - (a o))) exp (-3(n (n (a+ (de -))))
(-B(n-
where WBinary is the original weighting of for everything: WLimear applies linear decay from for perfect vector match 0 for the maximum distance allowed between vectors; and WExp is an exponential decay weighting that uses a 3 coefficient for its decay rate.
As will now show, the linear weighting applies the most weight patterns right in the middle of the pattern and query with gradual, symmetric decline around this point. Mcanwhile, the exponen- tial weighting cancels out to apply constant weight re-scaling to everything. Letting represent the distance of neuron to the read and write vectors it is weighted by, x (n 2) (a+c) b-2 2. Because we want to know the weighting coefficient that applies to neurons at all possible distances from the pattern and query, without loss of generality we can set 0 for the analysis that follows:
((dg 2) x))
WLnnaar (a, c, dy, n)
n
71
(C
xz
n
n
n²
x²
nC + xC
n
- x x - C) Z
(10)
n2
This formulation was first inspired by Jacckel
1989h
24

---

Published as a conference paper at ICLR 2023
where C. = (d. +2) is constant is Z ""=n"" We can take the first and second derivatives of Eq- 10 to know that the most weight is applied at the distance right between the read and write vectors, C72. This linear weighting applies different weights to neurons different distances. As result, the patterns stored in superposition will also have different weightings. However, empirically, this has negligible effect on the SDM exponential approximation, as shown in Fig. 7 We hypothesize that this is due to two factors: (i) the difference in weight values is not particularly large; (ii) neurons receiving the largest weights are the most numerous. Therefore, the weighting correlated with the approximately exponential decay in the number of neurons that exist in the circle intersection as vector distance changes.
As for the exponential, it cancels to constant term that depends upon the choice of (3:
WExp(a, (a, c, du n) cxp ( 5x) exp (- B(C -vx) x))
exp ( ( BC).
(11)
This constant term modification to the weighting of all patterns is then removed by the normalization term in the softmax operation, resulting in no effect on the output.
We confirm our results empirically for the three optimal SDM hamming distances d' and n = 64
dimensional canonical Transformer Attention setting used throughout (Bricken & Pehlevan 2021 in Fig.
Nommolized Weight for Circle Intersections
Nomollzed We ght Circle Intorsections diye = 15, r
diye 5, /1244, n=
0.2
0.30
Liner
Binary
Exp
Exp
0.23
0.-
Nemalaed Whnggtt
0.4
0.35
0
0.0
0.05
0.0
Hamming Distance a., batwocn vectors
Hamminn Distance d,,. betwoon two vectors
(b)
Normelized Weight for Circle intersections diss 25, 6664, r=/)²
0:30
ainary
Linear
0.08
Nomaatzad Welght
0.06
00
0.44
0.00
10
50
Hemming Distence du bet ween two vectors
Figure 7: Weighting neuron read and write operations has a negligible effect on SDM. The Exponential weighting (green) as expected gives the same result if there was Binary weighting (blue) making overlap perfectly. The Linear weighting (orange) results in a negligible difference that is barely visible due to high overlap.
25

---

Published as a conference paper at ICLR 2023
B GABA SWITCH
B.1 BIOLOGICALLY IMPLAUSIBLE SOLUTIONS TO THE DEAD NEURON PROBLEM
The dead neuron problem is where neurons are never activated by any input data and are waste of resources. This is particularly a problem with the Top-K activation function, especially at initial- ization where it is possible for small subset of neurons that happen to already exist closest to the data manifold to always be activated and in the Top-K. This means that none of the other neurons are ever activated, and thus never receive weight updates, preventing them from learning the data manifold and becoming useful.
The first set of strategies to solve the dead neuron problem initializes neurons immediately on or near the data manifold, for example, using PCA or SVD (Rumelhart & Zipser 1985; McInnes & Healy, 2018). Biologically, this would assume that priors on the location of the data manifold are encoded in our genome and pre-determine the structure of neuronal dendrites. However, dendrites often form in a highly dynamic and stochastic way (Dhar et al. 2018) Purves et al.) 2001).
In addition, this solution requires the genome store large amount of information and would limit an organism's adaptability to learn new data manifolds during its development, for example by moving to new environmental niche. It is popular in deep learning initialize weights not using the data directly but with optimized strategies such as Xavier (Glorot & Bengio (010) that empirically work well, even for the sparse ReLU activation function. However, as evidenced by the continued development of new initialization strategies, this approach remains heuristic and does not generalize across datasets, networks, and training techniques (Aguirre 2019 Chang et al. 2020b
Goodfellow et al. 2015).
The second set of approaches to solve the dead neuron problem ensures that all neurons, independent of their initialization, are active and update to learn the data manifold. This is major cited reason why ReLU is often avoided (Goodfellow et af.) [013 Henddyykks & Gimpel| (016),, even though in practice it often does not seem to significantly harm performance (Pedamonti, 2018 Trockman & Kolter 2022). The Top-K model of (Ahmad & Scheinkman 2019). VQ-VAE (van den Oord al. 2017), and Switch Transformer (Fedus et al., (222)) all address their dead neuron problems in biologically implausible ways. When using the Top-K activation function, the work of (Ahmad) Scheinkman 2019) artificially "boosted" the ranking for each neuron be in the Top-K, with more inactive neurons getting higher rankings. This solution makes the biologically implausible assumption that neurons fire function of how inactive they and undergo ubiquitous anti- Hebbian plasticity. Mcanwhile, the VQ-VAE and Switch Transformer both used terms in their loss function increase the utilization of "dead" code vectors and expert modules, respectively.
B.2 GABA SWITCH BIOLOGY
We summarize how the GABA switch works biologically in Fig. g (Gozel & Gerstner 202
Neurons are excited by GABA early in development before being inhibited by it due to changes in intracellular CL. concentration. After neurogenesis, neurons express NKCC1 which imports CL. into the cell causing the CL. reversal potential to be more positive than resting potential (Gozel & Gerstner 2021 Heigele al. 2016). When GABA is present, CL" flows towards its reversal potential, resulting in depolarization. Over time, this GABA activation indirectly results in increased KCC2, which instead pumps CL. out of the cell (Ganguly al. 2001; Connor et al. 1987). This makes the CL reversal potential more negative than the resting potential and hyyper-olarization during GABA activation.
This biological mechanism suggests that our GABA switch implementation in Eq. 12 could be made more sophisticated. Rather than counting up the number of times the neuron is simply active (a binary outcome), it could switch as function of the actual activation amount that the neuron experiences or size of gradient updates.
26

---

Published as a conference paper at ICLR 2023
1. Neurons swap out ion transporters from more activity
2. This causes GABA to transition from excitatory to inhibitory
Young
CI
crosses the neuron resting r potential
Young
Neuron in activated many times.
Membrane Potential, V/8
Naurnn Rlssinng Posental
Transporters:
CI
GABA
GABA excitatory
imports
inhibitory
KDC2: exports Ci-
Figure 8: Summary of GABA Switch Biology. Here visualize the ion transporters inside young versus adult neurons and how they change over time. The left side of the figure shows how neuron gets activated more times swaps out the concentration of its ion transporters from NKCC1 (blue) to KCC2 (red). This causes the neuron to go from importing chloride ions (CI-) to exporting them. The right side of the figure shows the consequence of this transition where the reversal po- tential of chloride goes from being more positive than the neuron's resting potential resulting in GABA being excitatory more negative resulting in GABA being inhibitory.
27

---

Published as a conference paper at ICLR 2023
B.3 GABA SWITCH PRACTICAL CONSIDERATIONS
There are three conclusions from this section on when and how to use the GABA switch to avoid dead neurons, improving continual learning:
If the positive weight constraint is being used then annealing k gives the same performance as the full GABA switch. If you allow for negative weights then the GABA switch should be used.
The subtraction operation works much better for continual learning than masking, How- ever, the inverse is true in a non continual learning setting. We provide some initial analysis of why this is the case.
The value of & for the GABA switch should consider the complexity of the data manifold, optimizer, and learning rate. When in doubt there is no harm setting s larger aside from requiring more training epochs.
We first give the full GABA switch algorithm that was developed and is used for the SDMLP throughout the paper. We then outline why the k annealing approximation works just as well when we constrain our weights and inputs to be only positive. Next, we discuss why subtracting to enforce Top-K K instead of masking leads to better continual learning performance. We conclude with other considerations on how to avoid dead neurons with interactions between the data manifold, learning rate, and hyperparameter
The GABA Switch Algorithm
Formally, the GABA switch algorithm is implemented as:
a = |a; - A+
descending sort cort([a)+ (())+)
A = min (1, 1a, ( 1, 1, + 2C1/s))
(12)
This algorithm is slightly more complex than the K annealing one presented in Eq. of the main text. C; is a counter for each neuron, recording the number of times that it has been activated since the start training. A; linearly increases from -I to I as a function of C. and [+] is the ReLU operation. is hyperparameter that determines the number of activations required for this neuron to switch from being excitated by GABA to being inhibited. When training on top of the ConvMixer embeddings, we found that 250, 000 ensured all neurons could move onto the data manifold, resulting in none being dead. As an example, if we assume all neurons are activated before the GABA switch, with this value the switch occurs after just 2.5 epochs of training on CIFARIO (there are 50,000 training examples). However, this value depends significantly upon the learning rate and complexity of the manifold where the case of training on raw CIFARIO images, to ensure there are no dead neurons we set s = 5, 000, 000 requiring 100 epochs for the GABA switch to occur. However, in this case we did not try to find the lower bound on s.
Why l annealing works just as well as the GABA switch when weights are positive
Because of Dale's s Principle, we implement SDM with positive weights and inputs. Empirically, this also boosts continual learning performance and introduces weight sparsity [13 that would increase computational efficiency if unstructured sparsity can be taken advantage of (Wang 2020).
When positive weights are used, we found no change in the number of dead neurons or continual learning performance using Eq. 4 linear annealing. This was true for both the ImageNet32 Con- VMixer embeddings and raw pixel datasets. However, when negative weights are allowed this was no longer the case and only the full GABA switch algorithm avoided dead neurons. We speculate that this is because when GABA is excitatory, the full algorithm will inject positive activation into neurons that may otherwise have negative activity and not fire thus failing to get a gradient update. Meanwhile, with positive weights and inputs the activation for every neuron is always positive sulting in every neuron being active unless Top-K K is enforced. Thus the positive weights and inputs
13 We found that ~ 20% of weights in two models checked (trained on the ImageNet32 Conv embed- ding and directly pixels) had values less than 0.01 meaning they could likely he pruned, however, we leave further investigations of sparsity including the introduction of T.' losses, to future work.
28

---

Published as a conference paper at ICLR 2023
Linear Annealing Avaids Dead Neurons only with Positive Weights
Linwar Arneal Negative Reightss GABA, Swi tch
ivve Weights
Lineer Arneal Positive Reights
GABA Switch Posi Weights
02
0.15
0.05
Ecocls
230
Figure 9: Linear Annealing needs Positive Weights. All lines overlap at the bottom of the plot aside from Linear Annealing when negative weights are allowed (green). The GABA Switch excites neurons that can otherwise be inactivated due their negative weights. This ensures all neurons update onto the data manifold and are not dead. However, linear k annealing presented in Eq, does not excite leaving these neurons dead. Training is directly on the CIFAR10 manifold which more difficult to learn than the ConvMixer embeddings.
ensure that every neuron receives gradient updates and moves onto the data manifold. In Fig. 9 we present this result training directly on CIFARIO pixels, a more difficult manifold to learn than the ConvMixer embeddings.
Note that while positive weights allow for linear & annealing, can also result in all weights be set to O, leading dead neurons and model training to fail.
Why subtracting works better than masking for Continual Learning
Empirically, masking results in much higher validation accuracies during pretraining than subtrac- tion (Fig- [Ob].. However, during continual learning, subtraction works better as shown in the ablations of Table 2 and Fig. [Oa] To work out why subtracting is better for continual learning analyzed the learning dynamics of both in App. H and discuss them here. We leave an analysis of why masking works better during pretraining to future work as do not care about maximizing validation accuracy for our pretraining task here.
Top-K K with subtraction utilizes more neurons during continual learning as shown in Fig. 10c This result supported by App. H Fig. 23 where there are fewer neurons that are consistently activated and they are more polysemantic, being used for multiple tasks.
Motivated by the stale momentum results of App. D we wondered if the larger activation values preserved by masking may lead more dead neurons when using SGDM as our momentum based optimizer. However, using SGD did not lead to any change in results and in hindsight our gradients being less than means that the results in App. D hold independent of the gradient values.
This means that the gradients must actually be smaller for subtraction and lead to slower updating. At first glance, the subtraction operation just reduces the activity of the & neurons that remain active by a constant term. This will scale the size of each gradient, equivalent to modifying the learning rate.
However, the situation is more complex. The activity of the k neurons and their learning rate is conditioned upon the activity of the k 1th neuron. If this neuron is almost as active those in the l subset, all of their activities will be very close to zero. Meanwhile, if the & + 1th neuron is much less active than the subset, their activities will remain large and they will receive a larger gradient update. These two situations correspond to the input being in densely and sparsely tiled regions of the data manifold, respectively.
We can think of this situation as approximating Bayesian one where we have mixture of Gaus- sians defining a posterior distribution over the data manifold. The number of neurons in a region is inversely proportional to the variance in the distribution at that location and thus changes the amount the likelihood our current data point updates the distribution. In other words, the subtraction op- eration introduces dynamic learning rate.
29

---

Published as a conference paper at ICLR 2023
Top-K Subtracting beats Masking for Continual Learning
Subtraction Masking
0.6
allddtiion accuracy
0,4
0.3
0.2
0.1
epoch
10k
Top-K Masking beats Subtracting for Pretraining
Top-K Subtracting Results in fewer Dead Neurons
Subtraction
Nasking
Subtraction
Masking
Frection of Deed Neurons
0.66
Veldatin Accurecy
0.25
0.5
0.1
0.2
0.05
0.1
Epochs
(b)
(c)
Figure 10: (a) Subtraction beats Masking for Continual Learning. However, as shown in (b) worse for pretraining. During pretraining. neither method creates any dead neurons. (c) During continual learning, across each task, there are fewer dead neurons when subtracting that may explain the better memory retention of previous tasks. Because these plots are not crowded, error bars show the min and max range across the five random continual learning data splits instead of the standard error of the mean.
During continual learning when the task changes and subregion of the data manifold is pre- sented, we want those neurons closest to the subregion and only those neurons to update and move onto it. In our stochastic optimization setting, it is possible that neurons not necessarily closest new subregion are updated towards instead, leading to an over-allocation of neurons to the new task. We hypothesize that the subtraction operation helps avoid this problem by making the learning rate both lower and dynamic. However, we believe the dynamic learning effect is more influen- tial than stochasticity because even when increasing the batch size from 128 10,000, we see that subtraction is still more robust to catastrophic forgetting.
likely that the reduced and dynamic learning rate is a doubled edged sword where while is better at remembering, it is also worse at learning new information. Both in pretraining and within each data split, the subtraction achieves worse training and validation accuracies. Why the subtraction operation does worse during the pretraining
single task we leave future work.
As an interesting aside, the Top-K subtraction operation implemented by an inhibitory interneuron when k 1 is the same Vickrey second price auction used in (Chang et al. (020a) to model neurons as individual agents with their own utility functions. The winning neuron places a "bid" represented by its activity amount. has its activity subtracted by that of the second most active neuron (thus paying the second highest price) and its remaining activity determines its exposure to the next gradient update.
Other GABA Switch Considerations Data Manifolds, Optimizers, and Learning Rates
Note that for the ConvMixer embedding, dead neurons not much of problem and either the GABA switch or & annealing can avoid any dead neurons within just a few epochs and with without positive weights. This is because the manifold is much easier to learn it is lower
30

---

Published as a conference paper at ICLR 2023
dimensional both in the size of vectors (256 versus 3072) and the data manifold being far more meaningfully correlated.
Fig. I shows how learning rate that is too low will cause SGD to produce dead neurons. The number of neurons dying being inversely proportional to the learning rate and the sudden jump in dead neurons before it plateaus just after the GABA switch both support the theory that the learning rate is too low. This means that neurons do not update quickly enough onto the data manifold before the GABA switch occurs. This sudden jump and plateau in dead neurons is in contrast to Adam and RMSProp optimizers shown in Fig. 15 where neurons will continue dying over time due to the stale momentum problems (Kingma & Ba 2015 Geoffrey Hinton 2012). Meanwhile, SGDM does not suffer from Stale Momentum problems to the same extent and is much more robust to choice of learning rate making it a good choice for pretraining. However, it still suffers somewhat from the stale momentum problem in the continual learning setting as shown in the ablations of Table 2.
To avoid dead neurons when using SGD either the learning rate can be tuned, the GABA switch value can be increased, sparse optimizer that avoids the stale momentum problem can be used (however, the success these sparse optimizers remains to be validated in future work). Also keep in mind that the learning rate chosen will be affected by the L normalization.
SGD with a low learning rate results in dead neurons
1 1r=0.2
D 1r=0.1
1r=0.00 09
1r=0.03 03
0.4
0.3
0.2
0.1
epoch
0
0
50 100
150
200
250
300
The GABA switch needs to account for the learning rate. We train SDM directly on CIFARIO pixels test its ability to Iearn the manifold and avoid dead neurons. The GABA switch occurs at epoch 1500 and each line denotes a different learning rate "Ir". For reasons outlined in App. D SGD should result in the fewest dead neurons. However, the learning rate needs to be carefully set in relation to the GABA switch threshold, s. the learning rate is too small, neurons won't update onto the data manifold quickly enough and will die instead.
Figure 11:
C TOP-K
C.1 INHIBITORY INTERNEURONS APPROXIMATE TOP-K
At a high level, inhibitory interneurons exist to regulate the firing of excitatory neurons, introducing sparse firing and keeping only the most active on This makes the brain likely to be highly sparse in the number of neurons firing at any given time. Estimates suggest that "an average neuron in the human brain transmits a spike about 0. 1-1 times per second." (Impacts) (022). while action poten- tials and refractory periods happen Assuming neurons fire randomly within this 10ms time interval, this gives back of the envelope calculation that a maximum of 10ms/100ml aligns well with the prediction of 15% by (Atwell & Laughlin, 2001). Sparse firing also makes sense from the perspective of metabolic costs where action potentials are expensive, consuming
time interval of roughty 10ms (Sterling & Laughlin] 2015).
10% neurons will fire within the time interval. This
"Inhibitory interneurons can also inhibit each other resulting in disinhibition but this is still used to perform sophisticated forms of excitatory neuron regulation.
31

---

Published as a conference paper at ICLR 2023
~20% of the brain's 's energy (Sterling & Laughlin 2013 Attwell & Laughlin 2001: Sengupta al. 2010).
However, in practice, is unrealistic to assume that inhibitory neuron can keep exactly A neurons active for any given imput because its afferent and efferent synaptic connections are hetcrogencous. The inhibitory interneuron sums together the activations from all neurons, weighted by their presy- naptic strengths and then outputs an inhibitory value that is scaled by the strengths of post synaptic connections. The weighted summation of inputs to the interneuron removes information on how many neurons are firing and at what rate. The heterogenous post synaptic connections will weight the effects of inhibitory on each neuron differently.
Keeping these concerns in mind, there are few reasons that we draw from for optimism that Top-K can be approximated. First, inhibitory horizontal cells the retina compute the mean activation value of thousands of nearby cone photoreceptors and inhbits them so that they encode contrast (Chapter 13 of (Sterling & Laughlin 2015).. Not only do horizontal cells have carefully tuned weights for their gap junction connections to compute this mean, but also will dynamically change its pre and post synaptic connectivity strengths to accurately compute mean activity under different lighting conditions. In low light setting, the horizontal cell will average over a larger number of neurons to reduce the variance in its estimate of the mean (page 252 of (Sterling & Laughlin (015)). If inhibitory interneurons in general come close to the level of sophistication shown by horizontal cells calibrating their synaptic connectivity, then possible the interneuron can compute how much it should inhibit to implement Top-K reliably.
Second, looking at biological evidence of Top-K from cerebellum- structures, there is strong evidence of Top-K being approximated in the Drosophila Mushroom Body by its Golgi interneuron analog (Lin et al. 2014). In mammals, the evidence is more complicated with some papers finding dense granule cell activations (Giovannucet et al.) 2017). However, other experimental evidence and theory suggests that granule cell sparsity in the mammalian cerebellum may be a function of task complexity (Lanore al. 2021 al., 2022). The lower complexity the task, the more dense the representations can afford to be as there are fewer stimuli that must be unique encoded. Meanwhile, high complexity tasks require larger number of orthogonal codes resulting in the need for sparser representations that lose fidelity firing remains an open question in need lower latency voltage indicators and the ability to record from more neurons simultaneously across more diverse range of tasks (Lanore al. 2021
result (Xie et al. 2022). Ultimately, the density of granule cell
Fraction of Active Neurons per Input
Mean Neuron Activation During Convergence
0.16
6-
Number of Neurons
5-
Activation Value
0.12
4-
3-
2-
0.04
0.02
0.00
0.14
0.16
0.18
0.20
0.2
0.24
0.26
0.28
250
500
750
1000
1500
1750
2000
Fraction of Active Neurons
Iterations
(a)
(b)
Figure 12: Inhibitory Interneuron Approximates Top-K. We reproduce the Hebbian learning rule of (Gozel & Gerstner 2021). (a) We show that it approximates having k Rs 200 of the 1000 neurons active for 50 different MNIST inputs. (b) We show an example of the average neuron activity running the ODEs given Eq. 13 until convergence.
Finally, tried to model the ability for inhibitory interneuron to approximate Top-K under the simplifying assumption that its pre and post synaptic weights are fixed and homogenous. We first analyzed the & neurons still on using the Hebbian learning dynamics of (Gozel & Gerstner 2021). We then looked at our SDMLP during learning and the relationship between the total sum of excitatory activations and the 1-th inhibitory value used in Eq. 4o keep only i neurons still
32

---

Published as a conference paper at ICLR 2023
The work that introduces the GABA switch (Gozel & Gerstner 2021 uses Hebbian learning rule with an inhibitory interneuron, An MNIST digit is presented for the network to learn and it obeys the following ordinary differential equations to determine the activities of the excitatory and inhibitory neurons:
c² W x
1" 0
Pa = c + WIEF f-1
1 e" + W -WEee'
P.
1
ga ( eee-1 +tanh (((p).
Ta
1
1 11
+Ipp b,]l+)
gi 7
e'
e 1-1 Ag.
Ag
(13)
where e is a vector describing excitatory neuron activities, the inhibitory neurons, and A is the learning rate. The excitation and inhibition time constants are Ta and 7ú, respectively. L is smooth- ing term, b is firing threshold learnt for each neuron. WI and WEr are the weight matrices from excitatory inhibition neurons and vice versa; they are fixed and have homogenous weights. Winp are the weights that respond to the input x. b; is the firing threshold for the inhibitory interneurons that fixed scalar and can be thought of as resulting in a noisy approximate value. During training, we use Hebbian learning rule (not given) to train Winp mp and run the ODEs shown to converge to approximately & neurons remaining on.
These dynamics are simplifications because the weights between excitatory and inhbitory neurons are fixed and homogenous. WET ET is initialized where every excitatory neuron is connected to each inhibitory with a probability of 90% and strength of 1. Wre is initialized with the same connectivity probability but with strength of 1/(0.9r,) where ry is the number of inhibitory neurons. This makes the synaptic weights connecting inhibitory to excitatory neurons
in expectation.
Putting aside these simplifying assumptions, when we run the model and analyse the number of neurons that remain active, it approximately constant i neurons shown in Fig. T2a Given the inhibitory activation threshold b, approximately 20% of the neurons remain on for each input giving 200. We also show how the average activity of the excitatory neurons evolves over time in Fig. 12b to emphasize that the inhibitory interneuron does not have to make a single guess for how to correctly inhibit all but & neurons. Instead, this is dynamic process defined by the ODEs of
Eq. 13
We also looked for our SDMLP the is + 1-th highest activity value used by the inhibitory neuron to implement Top-K K and how this relates to the total unweighted sum of neuron activations entering the inhibitory interneuron. After the GABA switch when neurons tile the data manifold and form subnetworks, this relationship is largely linear where the k + 1-th highest activity value is the total sum of all neuron activations before inhibition, divided by the total number of neurons. This is again a simplification by assuming all input and output weights are homogenous and equal to one. However, again, if the interneuron can modify its pre and post synaptic strengths with the same degree of precision displayed by horizontal cells (Sterling & Laughlin 2015) then these results support the possibility of the interneuron approximating Top-K, K, using a simple function to respond to the sum of its inputs.
33

---

Published as a conference paper at ICLR 2023
C.2 OPTIMIZED ToP-K
The k value is parameter which must be chosen in our model and can significantly affect perfor- mance. We present empirical results for ablating & on both pretraining and continual learning across number of different data manifolds.
To summarize our findings:
A larger it gives better pretraining but worse continual learning and vice versa.
k depends upon the complexity of the data manifold and number of neurons,
SDM will approximately tile whatever data manifold it is given. Joint training of SDM with other model components such CNN will improve pretraining. However, learning a non static manifold in this joint training setting is difficult for SDM.
Optimal values for Pretraining or Continual Learning?
Sparsity in the SDM setting presents a fundamental tradeoff between performing well in continual learning (CL) and non continual learning (NCL) settings, This tradeoff exists because sparsity limits the representational capacity of the model and should be expected to reduce NCL accuracy. Meanwhile, the more sparse the model is, the better it is able to form subnetworks that are not overwritten by future CL tasks. Additionally, SDM will tile the data manifold it is given, which depending on the manifold, can be poorly correlated with maximizing classification accuracy.
Tradeoff between training and continual learning per k value
Pareto Frontier between training ng and continual leamning
0.75
magehees52 Split CIFARIC Learning
Accorac
0.5
0.50
0.55
0.4
0.-
0.45
0.40
18 25 300 250
0.125
0.150
a.775
3200
0.225
0,250
0275
0.300
volidation Accuracy
(a)
(b)
Figure 13: Two perspectives on how the & value influences the ability to learn the ImageNet32 pretraining task and the Split CIFARIO continual learning task. (a) Validation Accuracies for final NCL ImageNet pretraining (blue) and final CL Split- CIFARIO (orange) over all tasks as a function of k values. (b) The pareto frontier between ImageNet pretraining accuracy (x-axis) and CIFAR10 continual learning (y-axis). Each blue dot is different k value. This plot makes evident the better pretraining performance for k < 8.
We see this tradeoff in Fig. 13 that summarizes ablating k during pretraining on ImageNet32 em- beddings from the ConvMixer and testing on Split CIFARIO. As the value decreases, continual learning accuracy improves while pretraining accuracy declines. However, it is unclear why perfor- mance on the original ImageNet32 training parabolic, slightly improving for k: < 8. While we are indifferent to the NCL pretraining task here, using it as way for the neurons to specialize across an arbitrary manifold of real-world images, it is worth acknowledging the performance is worse. That this performance decline is caused by sparsity is more evident during single training run as GABA switches and & is slowly annealed. Fig. [4a shows this for smaller values where performance declines.
To emphasize how the optimal & value changes as a function of the number of neurons and data manifold, describe number of additional results:
For the same ConvMixer embeddings ImageNet32 pretraining and Split CIFARIO testing, if we use 10K neurons instead of IK then there is the same linear decline in pretraining performance as
34

---

Published as a conference paper at ICLR 2023
in Fig. 13 but k outperforms all & I 50, it is also the best for continual learning. Meanwhile, k = becomes the worst for pretraining and the second best for continual learning.
With the same setup again but operating directly pixels without using the ConvMixer embeddings, k 10 is the best for pretraining but k = is the best at continual learning as shown in Fig. 18 of
App.
In all of the above cases we are given a static manifold for SDM to learn. This always results in worse pretraining accuracy than an equivalent ReLU MLP, even when using large ls values. This is because the ReLU MLP is not constrained to the data manifold, having a bias term and no L normalization. This allows it to learn weights and biases that better maximize NCL. validation accuracy.
Validation Accuracy for ConvMixer with SDM Layer
Dead Neurons
Outputs
k=25
k=10
km250
00
2-
km18
0.8
0.5
0.5
0.2
0.2
epoch
spoch.
40
60
80 100
120
140
0 20
40
60 100
120
140
(a)
(b)
Figure 14: Sufficiently small & values harm network training. ConvMixers with SDM layer at the end trained on CIFARIO. All networks perform the same for the first 20 epochs at which point Top-K is approximately fully implemented (the GABA switch occurs around epoch 10). (a) Validation accuracy for all models. The smaller the k value, the worse accuracy is. (b) The fraction of neurons that not active for any of the validation inputs (this is an upper bound the true number of dead neurons that may be inactive for the training data). Training the CNN portion of the network along with the SDM neurons leads many dead neurons. We think this because the manifold is still updating. As more neurons die, the representational capacity of the model declines.
If we perform joint training of the ConvMixer and SDM module then the ConvMixer can learn to create a manifold for SDM to tile that does maximize validation accuracy, performing on par with the ReLU MLP. This what happens in a test where we train on the whole CIFARIO dataset in the NCL setting, as long k 2 250, but independent of if there are 1,000 10,000 neurons in the SDM layer. This result is shown in Fig. [4a and suggests that the ReLU network only needs at least 250 neurons in its final layer to backpropagate gradients and achieve high prediction accuracy. There is evidence that artificial neural networks are overparameterized at the start of training and functionally sparse, as supported by results from network pruning, most notably the Lottery Ticket Hypothesis (Frankle& & Carbin 2019).
However, this result is again manifold dependent whereby training instead on ImageNet32, even with k = 2, 500 and 10,000 neurons still harms performance compared to ReLU network. We believe this is because ImageNet32 has dramatically more complex data manifold with 1.2M images from 1,000 different classes, compared to 50,000 images in 10 classes for CIFARIO. This means that even le of 2500 too small and harms the model's 's representational capacity.
An additional difficulty with joint training is the timing of the GABA switch to avoid dead neurons. This because the manifold SDM learns is dynamically changing over time and we believe explains the neuron death that is inversely proportional to & shown in Fig. 14b
We leave to future work to further investigate optimal k values and joint training versus using frozen, pretrained models. We are optimistic that the best way to resolve the tradeoff from sparsity limiting representational capacity is to make the network layers wider rather than the number of neurons that can be active, k. fewer. This is feasible because sparse activations are computationally cheap- while computing the most active neurons requires multiplying the input with each neuron (this can be parallelized) and then sorting them, constant A << neurons produce outputs and receive gradient updates. Computing which neurons are active would also be cheaper with sparse weights (Jaeckel 1989a Ahmad & Scheinkman 2019.
35

---

Published as a conference paper at ICLR 2023
SDM Theory to derive optimal l values
Before concluding this section, we want to flag to readers that SDM, with a few simplifying assump- tions, is able to analytically derive optimal k values under few different conditions of optimality: (i) Optimising the signal to noise ratio of every pattern; (ii) Optimizing for the maximum number of patterns that can be stored, within certain retrieval probability; (iii) Maximizing the distance query can be from its target pattern while still converging correctly. These derivations and further discussion be found in (Kanerva 1993 Bricken & Pehlevan 2021).
While flagging that these analytical results exist and are interesting, there are all concerned with maximizing the information content of the patterns stored that is most relevant a reconstruc- tion task. In this paper, we care about two different objective functions somewhat uncorrelated with maximizing information content: (i) Classification accuracy, where better performance can be (ii) Continual learning, where while we care about modelling the data manifold, we also want the formation of unique subnetworks that can out- put the correct classification label. In addition, SDM's analytical results assume that the patterns are random. For correlated real-world datasets, we need ways to quantify the complexity of the data manifold and a full exploration of this problem and solutions is beyond the scope of this work.
achieved without trying to model the data manifold:
D STALE MOMENTUM
Imagine a neuron is activated for three training batches, then inactive for the next 10 batches, and finally activated two more times. During the first set of activations there are no problems, the neuron will update its weights and its momentum term, using up to date gradient information. During inactivity, problems begin as the inactive neuron will still update its weights and do so using an increasingly out of date momentum term. Problems continue when this neuron is activated again because the momentum term decays slowly and will significantly boost the gradient in an out of date direction.
These stale momenta are especially harmful in the Top-K competitive activation setting where neuron needs to be the h most active for at least one input to continue receiving gradient updates, otherwise it will permanently die. Using Stochastic Gradient Descent (SGD) that free from any momentum term removes this problem. SGD with momentum works worse than SGD without momentum, but still better than Adam or RMSProp (Kingma & Ba 2015 Goodfellow et al. 2015).
Geoffrey Hinton 2012
Fraction of Dead Neurons for Different Optimizers
Adam
RMSPrap
SGDM
SGD
Fraction Neurons
Fooch
200
The GABA switch occurs by epoch 100 and Top-K is fully implemented by 200 epochs. At this point, the RMS and Adam optimizers begin continuously killing off neurons such that they are never activated for any training or validation data. These results are for training directly on CIFARIO images. See the text for discussion of training on ConvMixer embeddings.
Figure 15:
We discovered the Stale Momentum effect when training our models directly on CIFAR1O images that have much more complex data manifold than the ConvMixer embeddings. Fig, 15 shows that after the GABA switch, when Top- K is fully implemented around epoch 200, Adam and RMSProp continue killing off neurons (Kingma & Ba 2015 Geoffrey Hinton 2012). These neurons dead
36

---

Published as a conference paper at ICLR 2023
in that they are never activated by any training validation data. Our results are robust across learning rates for the different optimizers.
While these dead neurons from Adam and RMSProp don't appear when pretraining on ImageNet32 embeddings, the Stale Momentum problem still lcads to incorrect gradient updates that harm con- tinual learning as shown in the ablations of Table 2. This is because fewer neurons are covering the data manifold and able to retain memories of separate tasks.
Notably, ReLU network trained with Adam or RMSProp also kills off up to 95% of its neurons with only minor effects on train and validation accuracy when trained directly on CIFARIO 10 pixels. SGDM and SGID again do not kill off any neurons for this network.
Dead Neurons L2 Gradient Magnitude
Neuron Gradient L.2 Magnitude
SCD
Adus
SQZM
RHSP oz
5GD
Aclar
5S0M
ReSProp
0.312
0.01
0.15
0.333
0.2
0.334
0:25
Step
3006
AUCG 2006 900
1108
200k
3006
4355
100. 500
(a)
(b)
Figure 16: Empirical Stale Momenta Seen During Training, (a) We take all neurons that are dead during batch and record the mean L magnitude that its weights experience during the update. Adam (teal) has the largest updates to dead neurons. Note that these magnitudes are still 5-10x smaller than the average for alive neurons. SGDM (purple) also updates dead neurons but to lesser degree. SGD (yellow) and RMSProp (orange) are not visible because they do not apply any updates to dead neurons. We start our x-axis at training step 300k because this is when there enough dead neurons for their gradient updates to be meaningfully calculated. (b) We track all gradient updates applied to ten random neurons during the course of training and pick representative to display here. We track gradient updates starting right as the GABA switch occurs around epoch 100 and all training steps from this point on are shown along the x-axis. RMSProp (orange) has the highest variance and magnitude in gradient updates followed by Adam (teal).
We logged the gradients of neuron weights during training to confirm that neurons are receiving gradient updates even while they are dead as shown Fig. [6a This figure initially confusing because, while Adam and RMSProp kill off the most neurons in Fig. [5] it is Adam and SGDM that
keep updating dead neurons in Fig. T6a Kingma & Ba 2015) Geoftrey Hinton 2012). Looking at every gradient update applied to a single neuron right after GABA switches in Fig. [6b provides
different perspective. Here, the highest variance and largest gradient magnitudes (as quantified by
their T,2 norm) produced by R.MPProp and Adam. These gradient spikes appear particularly in
cases where the neuron is dead for some time and then activated again.
To investigate further, we implemented each optimizer in a toy environment tracking a single weight that receives sparse gradient updates to see how the moving averages respond [5] We were able to reproduce the much larger gradient spikes displayed by Adam and RMSProp upon the first few gradient updates after a period of quiescence. These results are shown in Fig. 17a] where we show the Aopimizer term for each optimizer, independent of the learning rate. Initializing their respective terms shown in Eqs. 14 with zeros, we first introduce two gradient updates and then have periods of quiescence before injecting gradient updates.
In Fig. 17a we show four different gradient injection periods and that RMSProp has the largest response, followed by Adam and then SGDM, reproducing the gradient spikes seen in Fig. [66] Fig. 17beexplains these larger gradients by plotting the Adam and RMSProp numerator and denominators for only two gradient injection periods. Note in particular that for Adam, its numerator m, (dark orange) declines somewhat quickly, having decay of 3 0.9, however, its denominator / (light
Further details for this toy experiment can he found and reproduced in our github repository in the Jupyter Notebook in the notebooks/ directory with the filename StaleGradients.ipyn
37

---

Published as a conference paper at ICLR 2023
orange) has much slower decay of 32 0.999, staying large for much longer and causing the new gradient imputs to explode in size.
Finally, Fig. 17c quantifies how large the gaps in gradient update magnitude between the three optimizers and the actual gradient when we vary the time between gradient injections.
To summarize, believe is more the gradient explosions after quiescence rather than updating of neurons currently dead that causes Adam and RMSProp to kill off neurons as Fig. 15 The
denominators present in these optimizers, given in Eqs. [66 and 17 are blame because they such slow moving averages and inflate the sparse gradients.
The update rules for SGDM, Adam, and RMSProp as follows where we let À be the learning rate, 0g are the parameters at time f, 9t is the gradient, and = 0.9 8 = =09, f3 = 0.999, a 0.99 hyperparameters for the various algorithms with their standard values.
8: = - Aptimizer
(14)
ASGD = 9t
ASODM = -08-11 +
(15)
m7
Adam
(16)
Vor
m
m4 = 1- $1
Ut
- B2
mi = 8 m + (1 B1)gt
32 (1 B2)9
RMMSPrOP = 9t
(17)
Vtz
= =0= + (1-a)
Note that these results are robust for any gradient updates St < 1, otherwise SGDM has larger gradient spikes than Adam. However, this is never the case in our training regime because of the L normalization term for our weights and inputs which means that no weight or neuron activation value the case then predict that SGDM would have sufficiently
ever larger than 1. If this large jumps in its gradients to also result in dead neurons.
Practical Takeaways Below we give our reasoning for why SGD the best solution but needs careful hyperparameter tuning of the learning rate and GABA switch s. Sparse optimizers (Spa) are another alternative that should be investigated future work.
Because SGD has no momentum term it does not suffer from stale momenta and is the most prin- cipled solution. However, lacking a momentum term, SGD is also the most sensitive learning rate choice and can introduce either slow or poor convergence if it is too high or low, respectively. Setting the learning rate too low particularly problematic in our setting with the excitatory period of the GABA switch because neurons fail to receive enough gradient updates to data manifold, resulting in dead neurons. In Fig. of of App. B.3 we show how a learning rate that too low will result in dead neurons that never learn the manifold before the GABA switch occurs.
onto the
Given these hyperparameter tuning requirements for SGD, we recommend using SGDM to pretrain SDMLP while using SGD for continual learning. SGDM pretraining does not result in dead neurons as shown in Fig. [S and more robust to choice of learning rate. For example in this instance of SGDM CIFAR 10 pixels, varying the learning rate from 0.01 0.09 did not affect either
38

---

Published as a conference paper at ICLR 2023
Adam and RMSProp have much larger gradient spikes than SGDM
10
SGDM
Adam
RMSProp
Gradient value
4-
2
80
100
Gradient Update Steps
(a)
login Between
Adam and RMSProp my and Components
Actual Gradient by Gradient Time Interval
10
0.30
10P
0.30
Isa.. Gindent Data
0.
10:1
Adam (92
RMSProo
RMMSPrap (Vi
Adam
0.9
RMSprrp
0.0
10
SGDM
40 60
20
40
60
Grawceent Update
Soans tinccient ixpection Interval
(b)
(c)
Figure 17: Toy experiment of how Adam, RMSProp and SGDM introduce stale momenta. (a) Gradient updates of 0.2 are injected at four different time points visible by the large RMSProp green dots. We first apply two gradient updates, then four, then one and one. Note how RMSProp am- plifies the gradient magnitudes but does not apply any gradient update when the neuron is inactive. Meanwhile, Adam (orange) and SGDM (blue) will keep applying gradient updates that slowly de- cay time. Also note that the Adam gradient magnitudes are larger than those for SGDM. (b) We inject gradients at single time points twice and observe how the numerators (darker color) and denominators (lighter color) for Adam and RMSProp change. The denominators decay very slowly. (c) We see how the delta between the actual gradient value of 0.2 and the gradient value applied by the different optimizers changes depending on the interval between the last gradient update. Note that the y-axis is log1n- RMSProp has 10/0.2 =50x amplified gradient and Adam is 1/0.2 = 5x. Note that SGDM without denominator term is the only optimizer that over longer periods of time stops amplifying the gradient value. All results generalize for gradient values that are < 1.
convergence speed or kill off any additional neurons. Meanwhile, as shown in the ablations of Table 2, in the continual learning setting. SGD is the best choice.
Another alternative are sparse optimizers like "sparse Adam" which does not apply gradient updates to any neuron that is dead (Spa). However, failing to update the momentum term in novel way will theoretically still result in what we believe to be the larger problem of exploding re- re-activated neuron gradients. The only way leave it to future work to empirically test sparse momentum based optimizers.
sparse Adam in Pytorch is to implement sparse weight layers and
unaware of existing literature around other mod- els in the sparse activation regime, using either Top-K or Mixture Experts (Fedus et al. 202T] Roller al. 2021 Shazeer et al. 2017) that discuss the stale momentum problem and issues with
This section lacks many citations because
39

---

Published as a conference paper at ICLR 2023
momentum based optimizers We also could not find academic citation for the sparse Adam implementation provided in Pa Porch or the motivations for implementing it. Suggestively, in Tensor- Flow the algorithm is called "LazyAdam" and advertised being advantageous for computational efficiency reasons without any mention of dead neurons. This is unsurprising in light of the fact that the dead neuron problem is only issue in the sparse regime and even in this case can go unnoticed if validation accuracy is the only metric of interest. For example, training on CIFARIO pixels with ReLU network and Adam results in 95% dead neurons with minor effects validation accuracy.
16
E
TRAINING REGIME ABLATIONS
E.
TRAINING DIRECTLY ON IMAGE PIXELS
Pixel Training Split CIFAR10
SDM k=1
SDM+EWC
EWC
SDM 10K
- SDM+EWC 10K
0.4
ation Accuracy
0.3
0.2
0.1
Epochs
2k 4k 6k
8k 10k
Figure 18: Rank ordering of continual learning algorithms remains approximately the same for training directly on CIFARIO. Here SDMLP with 10K neurons (yellow) actually does better than SDMLP+EWC (green) that the best performer in the main text Fig. aa). However, not tune the hyperparameters for the EWC loss coefficient or the 3 parameter for training directly pixels. SDMLP+EWC with 10K neurons fails to terminate within the 15 hours of GPU time allocated however it is clear that SDM outperforms it. Results are for the random seed splits of Split CIFAR 10.
did
We remove the ConvMixer preprocessing and train models directly on ImageNet32 pixels before testing their continual learning abilities on Split CIFAR 10 pixels. This is way to address concerns that the ConvMixes preprocessing was manipulating the CIFAR1O data manifold to make the contin- ual learning task too casy. Fig. 18 shows the results and Table 3. We only tested the best performing models with reference to Table and chose EWC over MAS because it gives the best model overall and otherwise performs similarly. We did not test the FlyModel because assumes the presence of preprocessed latent embeddings rather than the 3072 dimensional CIFAR 10 flattened image vectors. All training settings and parameters are kept the same in the Table tests.
Interestingly, all EWC methods fail to perform as well as they did before. This is likely because we did not tune loss coefficient 9 parameter for training directly pixels. As result, the best performing model is now SDM with 10K neurons (yellow) instead of SDMLP+EWC (green). Also note that while SDMLP with & 1 (pink) does worse, fails learn each task within the 2,000 epochs, e.g. at the end of 2,000 epochs the validation accuracy is ~00% while the other methods maximize the accuracy within task of ~20%. This means that SDM with & could potentially do better with more epochs or higher learning rate.
16 "nnterestiigly, (Sazzer et al. 2017) modifies Adam to reduce its parameter count by setting and taking averages ft, both of which may have inadvertently helped avoid stale momenta.
40

---

Published as a conference paper at ICLR 2023
We use the same 5 random seeds and 2,000 epochs per task. Training takes longer because the im- ages are 3x32x32=3072 pixels rather than the 256 ConvMixer embeddings and the most complicated model. SDMLP+EWC with 10K neurons fails to terminate in the 15 hours of GPU time allocated to each run but makes it to the last learning task where it's performance can be inferrred.
Table 3: Pixel Training Split CIFAR10 Validation Accuracy
Method
Neurons
k
Val. Acc.
SDMLP
IK
1
0.33
SDMLP
10K
10
0.42
EWC
1K
NA
0.19
SDMLP+EWC
1K
10
0.28
SDMLP+EWC*
10K
10
0.32
Oracle
1K
NA
0.53
Oracle
10K
NA
0.53
The most competitive models, pretrained on ImageNet32 and tested on Split CIFAR1O without any ConvMixer embedding. SDMLP+EWC with 10K neurons has to denote that it did not finish training. Average of 5 random task splits.
41

---

Published as a conference paper at ICLR 2023
E.2 No PRETRAINING
No Pretraining Split CIFAR 10
SDM k=1
SDM+EWC
EWC
SDM+EWC
10K
SDM 10K
Validation Accuracy
0.2
Epochs
2k
4k
6k
8k
Figure 19: Rank ordering of continual learning algorithms remains for no ImageNet32 pre- training. Solid lines denote different algorithms. Dotted lines denote variants of an algorithm. SDMLP+EWC (green) with 10K neurons does the best then SDMLP with 10K neurons (yellow). The 1K algorithms all do approximately the same including EWC (purple), SDMLP+EWC (dotted brown), and SDMLP k=1 (pink). Results are for the 5 random seed splits of Split CIFAR 10. TopK and ReLU benchmarks both do poorly and were not run for the full set of random seeds or shown as result.
Fig. 19 shows the results for the most competitive continual learning algorithms without any Im- ageNet33 pretraining and tested the ConvMixer embedded Split CIFAR1O. See Table 4 for the final validation results.
We train for 2,000 epochs each task and 10,000 epochs in total. The GABA switch occurs well within the first task where & 5, 000, 000 with GABA switching at epoch ~ 50 and Top-K being fully implemented by epoch 100. Performances are lower than with the ImageNet32 pretraining, for example the best performing SDMLP+EWC is 84% versus the 86% in Table Most notable the worse performance for the 1K neuron models that lack sufficient capacity to learn the data manifolds of new tasks without the benefits of pretraining.
Table 4: No Pretraining Embedded Split CIFARI0 Validation Accuracy
Method
Neurons
k
Val. Acc.
SDMLP
IK
1
0.56
SDMLP
10K
10
0.77
FlyModel
IK
32
0.69
FlyModel
10K
0.82
EWC
IK
NA
0.52
SDMLP+EWC
IK
10
0.54
SDMLP+EWC
10K
10
0.84
Oracle
IK
NA
0.93
Oracle
10K
NA
0.93
42

---

Published as a conference paper at ICLR 2023
F ADDITIONAL DATASETS
F.I CIFARIO EXTRA FIG. AND TABLE
Table 5 shows the Split CIFARIO results for the IK neuron setting. Fig. 20 shows the extent to which the SDMLP forgets previous tasks when compared to the baseline ReLU model. This figure gives extra context the headline continual learning results of Fig.
Table 5: Split CIFARIO - 1K Neurons Validation Accuracy
Method
Neurons
k
Val. Aec.
SDMLP
IK
1
0.70
SDMLP
IK
10
0.63
Top-K
IK
10
0.29
FlyModel
1K
0.70
MAS
IK
NA
0.69
EWC
IK
NA
0.67
SI
IK
NA
0.34
NISPA
~1
NA
0.19
L.2
1K
NA
0.23
Dropout
1K
NA
0.21
SDMLP+ M
1K
10
0.83
SDMLP+EWC
IK
10
0.83
Oracle
1K
NA
0.93
Split CIFARIO Final Validation Accuracy We highlight the best performing SDMLP, baseline,
and overall performer in the 1K neuron setting. Oracle was trained on the full CIFARIO dataset. Si has a denoting some of its runs failed. NISPA uses three hidden layer network with 400 hidden units per layer but this is close in parameter count to one hidden layer with 1K neurons. All results are the average of 5 random task splits.
SDM ReLU Forgetting Per Task
SDMLP T1
ReLU T1
SDMLP
ReLU T2
SOMLP T3
ReLU
Volidation Accuracy
0.5
0.4
Task #1
Task #2
Task #3
Task #4
Task #5
0,2
Epochs
4k
8k
10k
Figure 20: SDMLP gradually forgets previous tasks. Solid lines along the top show SDMLP forgetting previous tasks over time in comparison to ReLU that catastrophically forgets (dashed lines along the bottom). Validation accuracy is now computed within tasks and avoid clutter only show the learning curves for the first three tasks. Both plots use the average of 5 random seeds and error bars show standard error of the mean.
F.2 CIFAR100
To better assess the true memory capacity of our models, we use the same pretraining on ImageNet32 and then test continual learning 50 splits of C1FAR100, shown in Fig. 21 and Table 6. We allow
43

---

Published as a conference paper at ICLR 2023
for 500 epochs per task and 25,000 epochs in total. We used three random seeds instead of five, the same hyperparameters found for 10, and only tested the best performers from Table 1.
It is with CIFARI00 that the SDM models including the FlyModel and SDMLP+EWC really shine over the regularization baselines. In the IK neuron setting, SDMLP+EWC is the best performer with 42%. Meanwhile, in the 10K setting the FlyModel takes the lead getting 58% and SDMLP+EWC behind it at 51% against the oracle of 72%. The next closest baseline 40% for MAS. The large performance jumps in the FlyModel going from IK 10K neurons for CIFARIO and CIFARI00 highlight the importance of dimensionality for the fixed neuron addresses that perform a random projection. Interestingly, when looking the validation accuracy on Task 1 for each of the models, which shows how it forgets the task over time, SDMLP+EWC performs better than the FlyModel, however the FlyModel retains later tasks better.
CIFAR100 Continual Learning Performance
.... Oracle
FlyM SDMLP+EWC - SDMLP
MAS - EWC - Top-K
0.72
Valldatton Accuracy
0.6
Epochs
Task Validation Accuracy
FlyM PWWC
SDMLP MAS
EWC
Top-K
0.8
0.6
5k
15k
25k
Figure 21: Top Rank ordering of continual learning algorithms remains similar for CI- FAR100. The FlyModel with neurons (magenta) now outperforms SDMLP+EWC (green). The SDM algorithm does the next best (yellow) with MAS (blue) again close behind and then EWC (purple) and Top-K (orange) performing poorly. Bottom SDM methods are robust forget- ting. The first task of CIFAR 100 and its validation accuracy for this task over the entire course of training on 49 other tasks is shown for each method. IDML.+EWCC is the most robust forgetting (green), then the FlyModel (magenta) and SDMLP (yellow). It is interesting that the SDMLP+EWC forgets less for this first task than the FlyMedel. Top-K shows some retained memory but only small amount while MAS and EWC both catastrophically forget time within 5 tasks (2,500 epochs).
44

---

Published as a conference paper at ICLR 2023
Table 6: Split CIFARIO0 Validation Accuracy
Method
Neurons
k
Val. Acc.
Method
Neurons
Val. Acc.
SDMLP
IK
0.32
SDMLP
10K
0.43
SDMLP
IK
10
0.39
FlyModel
10K
32
0.58
Top-K
1K
10
0.11
MAS
10K
NA
0.40
FlyModel
IK
32
0.36
EWC
10K
NA
0.16
MAS
IK
NA
0.24
SDMLP+EWC
10K
10
0.51
EWC
IK
NA
0.12
Oracle
10K
NA
0.72
SDMLP+EWC
IK
10
0.42
We bold the best performing model within the 1K and 10K neuron settings. Oracle was trained on the full CIFARIO dataset. Average of 3 random task splits.
45

---

Published as a conference paper at ICLR 2023
F.3 SPLIT MNIST
For the sake of completeness and providing an easier benchmark for other methods, we use the Split MNIST dataset but keep the class incremental setting. Results are presented in Table and approx- imate the general rank ordering of performance whereby the FlyModel with 10K neurons does the best, SDMLP+EWC comes second, then SDMLP on its own and then the parameter importance reg- ularization methods. We did extensive hyperparameter tuning of the EWC and MAS regularization coefficients and parameter along with the FlyModel parameters (App. G.2
We do not use any pretraining and train directly on MNIST pixels. We train for 500 epochs on each task meaning 2,500 epochs total. We use just three random seeds initialize model weights but always use naive split where task contains digits 0 and 1, task 2 contains digits and 3, etc.
As one of our benchmarks we used Active Dendrites (lyer al. 2022). We used the code provided but found that it failed to generalize beyond the easier Permuted MNIST task incremental benchmark used the paper.
NISPA has in the number of neurons column of Table 7 because it used three hidden layers of 400 neurons each as in its original implementation. This is approximately the same number of parameters the IK neurons in single layer (637600 794000 ignoring bias terms) 17
Table 7: Split MNIST Validation Accuracy
Method
Neurons
k
Val. Acc.
SDMLP
1K
0.69
SDMLP
1K
10
0.53
SDMLP
10K
10
0.53
FlyModel
IK
64
0.77
FlyModel
10K
32
0.91
EWC
1K
NA
0.61
EWC
10K
NA
0.67
MAS
1K
NA
0.49
MAS
10K
NA
0.58
SI
1K
NA
0.36
Top-K
1K
10
0.22
NISPA
NA
0.40
Active Dendrites
1K
NA
0.20
ReLU
1K
NA
0.21
SDMLP+EWC
IK
10
0.83
SDMLP+EWC
10K
10
0.86
Oracle
1K
NA
0.98
Oracle
10K
NA
0.99
Models trained directly on MNIST pixels and without any pretraining. We bold the best performing method. We run these results on three random seeds and using 500 epochs for each split (this is much higher than other baselines (Hsu et al. 2018)).
Interestingly, the SDMLP with 1K neurons does better than with 10K neurons. We also found that the SDMMLP+EWC models were still learning the last task at the end training. However, increasing training times did not result in the last task being learnt better, suggesting that the model has run out of new neurons to avoid overwriting, or that the regularization coefficient for EWC should be reduced for the combination with SDM
NIIPA 80% weight sparse here but this aids its continual learning and do not consider the activation sparsity the Top-K models our parameter counts either. It also takes many more FL.OPs to train because of the iterative weight growth and pruning.
This effect also observed for the FashionMNIST results in App. F4
46

---

Published as a conference paper at ICLR 2023
F.4 SPLIT FASHIONMNIST
We also evaluate our most successful models and baselines on the FashionMNIST dataset that has different classes of fashion items grayscale 28x28 images. This task is slightly harder than MNIST with our oracles getting 90% instead of 99% for MNIST. Table 8 shows these results as the average of random seeds used to initalize the model weights. The rank ordering of results agrees with that of Split MNIST Table 7.
Table 8: Split NNSST Validation Accuracy
Method
Neurons
k
Val. Acc.
SDMLP
IK
0.73
SDMLP
IK
10
0.53
SDMLP
10K
10
0.52
FlyModel
1K
64
0.67
FlyModel
10K
64
0.76
EWC
1K
NA
0.68
EWC
10K
NA
0.72
MAS
1K
NA
0.33
MAS
10K
NA
0.34
Top-K
IK
10
0.23
ReLU
1K
NA
0.21
SDMLP+EWC
IK
10
0.74
SDMLP+EWC
10K
0.72
Oracle
IK
NA
0.90
Oracle
10K
NA
0.90
Models trained directly on FaSHionMNIST pixels and without any pretraining. We bold the best performing method.
47

---

Published as a conference paper at ICLR 2023
G BASELINE IMPLEMENTATIONS
G.1 BETA CORFFICIENT FOR ELASTIC WEIGHT CONSOLIDATION AND SYNAPTIC INTELLIGENCE
The paper (Hsu et al. 2018) considers a number of baseline continual learning algorithms with different continual learning settings. This includes the most realistic class incremental setting that we use. The paper also open sourced their implementations of these different algorithms and found that the regularization methods EWC, MAS and SI (Kirkpatrick et al. 2017 Aljundi et al. 2018 Zenke al. 2017) all catastrophically forget Split MNIST.
However, in our hands by more carefully tuning the loss coefficient for the regularization term in the loss function were able to improve performance. Tuning the loss coefficient for EWC and SI did not increase their performance. But when we looked at the EWC and SI learning dynamics, we found that because they were getting 100% accuracy on each task split, there was no gradient from the loss that could be used to infer the importance of each weight to be regularized for future tasks. In order to give the model gradient information, modified its cross entropy loss, introducing g < coefficient that made the model less confident in its prediction. Formally:
exp (BI)
P: E exp (5L))
(18)
where I E R are the real valued outputs (logits) for each of the o output classes, indexed by i. And when put through the softmax distribution this gives probability distribution where Σi=1 Pi
1. In the original algorithm of (Hsu et al., 2018). 3 = 1 by not being a hyperparameter to tune and this 3
is large enough that it results in the model output for the correct class getting probability of ~ with values of 0 for all other classes, giving a loss of 0 and no gradient. By having 3 < we force our model to be less confident in its prediction of the correct class, creating a loss and gradient information to infer which parameters are important. Note that this 3 term only used when inferring parameter importance, not when training the model. By tuning the 3 value in conjuction with the regularization loss coefficient, we were able to avoid catastrophic forgetting and again exceed the baseline reported in (Hsu et al. (0118) for EWC and SI. While this is another hyperparameter that must be tuned, and is not modification mentioned in the original algorithms of (Kirkpatrick et al. [0177] 7 Zenke et al. (017), or other literature that we are aware of, it is a straightforward modification that boosts performance and retains the original essence of the algorithm. Therefore, we feeling that it is an appropriate modification to create more meaningful baselines.
Table 9: Improved Regularization Baselines on Split MNIST
Name
Loss Coef.
B
New Val. Acc.
Original
EWC (Kirkpatrick et al.
2017)
200
0.005
63.23
19.80
SI (Zenke et al. 2017)
1500
0.005
35.76
19.67
MAS (Aljundi et al. 2018)
0.5
NA
24.99
19.52
1.2 (Goodfellow et al. 2014)
10
NA
36.77
22.52
Testing a 1,000 neuron single hidden layer MLP on Split MNIST with 10 epochs per split we get the final validation accuracies after hand trying few different hyperparameters shown in Table 9 We show the hyperparameters used and present results from the class incremental setting of Table in (Hsu et al., 2018) for comparison.
Note that the number of epochs here is fewer than in the full Split MNIST analysis of App. F3 This is to relate our results to other baselines that use only small number of epochs per task (Hsu al.) 2018). If you compare Table 9 with Table 7 of F.3 you will see that training for more epochs does affect the performance of EWC and MAS.
For our full experiments using Split CIFARIO in the pretraining setting shown in Table 1, we do more extensive Bayesian hyperparameter searches to maximize the performance of each baseline.
48

---

Published as a conference paper at ICLR 2023
G.2 FLDMODEL PARAMETERS
The FlyModel is unique in being trained for only one epoch and not using backpropagation. The authors outlined method for the model to be trained for more than one epoch each task, however, this introduces synaptic decay that would likely reduce performance via forgetting. It was never implemented in (Shen et al. (021) and so we present the strongest version of the algorithm trained for one epoch on each task here.
We used the parameters outlined in (Shen et al. 2021 starting point. Because the dimensionality of our ConvMixer embeddings is 256, half of the 512 used in the original paper, we were able to cut in half the number of Kenyon cells from 20,000 to 10,000, conveniently fitting the number of neurons considered in our other experiments. We also varied the number of projection neuron connections between 64 and 3, finding that 32 performed better in both the 10,000 and 1,000 neuron settings. We experimented with the learning rate and found that the value of 0.005 worked best for the MNIST and CIFARIO experiments. For CIFAR 100, we got better performance using a learning rate of 0.2 but only for the 10,000 Kenyon cell model. The best results for each run across the random seeds is what is presented in the text.
H INVESTIGATING DIFFERENCES IN CONTINUAL LEARNING ABILITIES
We investigated why SDM robust to catastrophic forgetting while baseline models and SDMLP models that ablate specific features all fail.
To summarize our findings:
ReLU- neurons never learn to tile the data manifold and every neuron is activated by almost every input.
Top-K the lack of L normalization and the use of a bias term means neurons don't 't tile the data manifold. This means they are not only activated by more tasks, forgetting previous ones. Additionally, there are more dead neurons reducing memory capacity.
SDM Top-K K Mask masking leads to more dead neurons and the neurons that are alive be- ing more polysemantic, failing to have subnetworks that avoid being overwritten by future tasks.
SDM No L Norm there are few neurons with massive weight norms that are active for up to 50% of all inputs. These neurons are active for all tasks, resulting in memory overwriting and catastrophic forgetting.
During Split CIFAR1O continual learning checkpoint model every 100 epochs, training on each data split for 300 epochs for a total of 1,500. We also track the number of times that each neuron is active across the entirety of training and visualize this on the logio plots shown in Fig. 22
Fig. 22a compares SDM to Top-K and ReLU where it is clear that ReLU neurons (green) are all activated many times without the Top-K activation function. While Top-K has fewer activations (orange), its neurons lack the bimodal distribution of SDM (blue) that we believe corresponds to the unique subnetworks only activated for a specific task. Fig. 22bllooks SDMLP ablations that use Top-K Mask instead of subtraction and no L normalization. Note the small blip of neurons for no L norm (green) that corresponds to the greedy neurons always the Top-K for all tasks. Also note that the Top-K K mask (orange) is slightly shifted towards larger activation values.
For all of our analysis that follow we do show dead neurons. These dead neurons are only really a problem for K that has 55% dead neurons and 18% for SDM without an 1,2 norm.
To look at how specialized each neuron to specific tasks and data classes, we take the final models after Split CIFAR10 continual learning and pass all of the CIFAR10 training data through them, recording which neurons are in the Top-K for each input. For each neuron, we count the number of times it is active for each of the 10 input classes and use this to create a probability distribution over the classes the neuron is activated by. We take the entropy of this distribution as a metric for how polysemantic each neuron is. We plot the entropy of each neuron against the percentage of time it is in the Top-K K in Fig. 23 We also combine these two metrics to weight each neurons' entropy by the amount of time it in the Top-K compute the average entropy of activated neurons and
49

---

Published as a conference paper at ICLR 2023
Hiszogram lngu:(number)
Histogram the logse(number)
70
SDM
SDM
400
Took
60-
Mys-
Number of Nourons
50
40
200
30
10-
3 (Numbrl of activations
6
3
7
(00u:ouume of activations
(a)
(b)
Figure 22: SDM Neurons are more specialized than Top-K or ReLU. We track the number of times every neuron is activated during continual learning and present this as a histogram with log1o scale. (a) We compare SDM (blue) to the Top-K (orange) and ReI.U (green) baselines. (a)
We compare SDM (blue) to two ablations, SDM with Top-K mask (orange) and SDM without T,2 normalization (green).
present this in Table 10 alongside the continual learning performance of each method. There is clear inverse correlation between the average entropy of activated neurons and continual learning 20
Table 10: Mean Neuron Entropy Weighted by Top-K Presence
Name
Mean Weighted Entropy
Val. Accuracy
SDMLP
0.99
0.54
SDMLP Linear Mask
1.36
0.35
SDMLP No I Norm
1.96
0.20
Top-K ReLU
1.48
0.29
2.25
0.21
Fig. 23 and the summary in Table 10 effectively convey the subnetwork formation by SDM that enables strong continual learning performance. SDM (Fig. (3a) has the lowest mean entropy of activated neurons followed by SDM with the Top-K mask (Fig. 23bb. The SDM with Top-K Mask Fig. 23b shows how the most polysemantic (highest entropy) neurons are also the most active which will result in forgetting across tasks. Fig. 23c strikingly shows 13 "greedy" neurons in the top right that are not only highly polysemantic but active for many inputs (compare the y-axis going up to 6% compared 1.4% for SDM). Top-K in Fig. 23d looks like the SDM with Top-K Mask but the average polysemantism of each neuron is much higher. Finally, the ReLU network in Fig. ?? while haved very even distribution of neuron activations, having the lowest y-axis range of 0.16, the neurons are all highly polysemantic.
Notably, the fact that SDM in Fig. 23a does not have every neuron activated and in the Top-K the same percentage the time means that it's neurons do not perfectly tile the data manifold in proportion to the density of the data. However, this may be because we optimizing the network for classification performance instead of reconstruction loss.
Note that in these experiments we used the SGDM optimizer instead of SGD. This means the validation accuracies for SDM and the linear mask are lower than they otherwise would be, however, we helieve the insights drawn here unaffected and SGD would have just resulted in fewer dead neurons.
50

---

Published as a conference paper at ICLR 2023
Entropies Percentage Top-K
J0M-LinearDeccyyMeek Entropies
Percentage Top-K
0.090
0.032
0.030
Arraatage time
0.008
of tnee
0.008
0.010
0.004
0.000
0.005
0.000
0.000
0.0
0.5
10
15
2.0
0.0
0.5
1.0
1.5
2.0
Neuron intrapy Over Data Llasses
Neuron Entrupy Over Data Claoses
(a)
(b)
Nol Neuran Entropies and Percentage
Top-K Nauron Entropies and Percentage Top
Tap-K
0.020
Percentapy time top-
the
0.015
0.03
0.010
0.005
0.01
0.00
0.000
0.0
0.5
1.0
15
2.0
0.0
0.5
1.0
1.5
2.0
Nouron Entrapy Over Data Classes
Neuron Entrogy Over Data Classas
(c)
(d)
Reil Neuron Entropies and Percentage of time in the Top-K
0.0036
Wivenaaee mn 1EE
0.0034
0.0006
0.0004
0,0002
1.6
1.7
1.9
2.0
2.1
2.
2.3
Niuuron Entrroy Over Data Classes
(e)
Figure 23: SDM Neurons specialized and participate democratically in learning. Plotting the entropy each neuron (the distribution data classes that the neuron activated by) against the percentage of time that is in the Top-K for all inputs.
51

---

Published as a conference paper at ICLR 2023
Training Weights
SOM-LinerDDecayMesk- Training
Weights
13
Cafa
12-
We ghts
11
UPM Din 2
10
9-
8-
7-
Data
Weights
6
10
UMAP Dim
UMAP Dmm 3
(a)
(b)
Training
Weights
Top-K. Training Weights
27.5
Data
25.0
22.5
20.0
15.0
3-
125
10.0
Data Weights
75
7.5
5.0
2.5
0.0
2.5
5.0
7.5
UHAP Dim
UMAP Dim 1
(c)
(d)
ReLU Training Deta on Weights
9-
8-
US6P Dm
Data Weights
11
-20
-9
-8
UMAP Dim
(e)
Figure 24: CIFARIO projected onto UMAP embedding of the SDM weights tiles the manifold the best. (a) is shown as Fig. 6 of the main text and the only approach that manages to tile the manifold such that the UMAP plot has meaningful structure.
52

---

Published as a conference paper at ICLR 2023
SDM Weights on Training Data
SDM-LinearDecayMMskk Weights Fit on Training Data
Fata
Cara
UMMPDmm 2
USG
12
-2.5
2.5
5.0
7.5
10.0
12.5
UMAP
UNAP
(a)
(b)
SDM-NoLZNorm: Weights Fit on Training Data
Top Weights an Training Data
10
Weights
URAP Dim
UNAP
12
14
UMAP
(c)
(d)
ReLU Weights Fit Treining Data
Weightt
UA6P Dm
2-
(e)
Figure 25: SDM Weights Tile the CIFARIO UMAP Embedding the Best. The two SDMLP models in the top row ((a) and (b)) do the best tiling of the manifold. This is unlike the previous Fig. [24 where only SDM in (a) showed good tililng. The CIFARIO data manifold looks different here for the top row, forming three distinct blobs because of the L normalization operation.
53

---

Published as a conference paper at ICLR 2023
We present number of additional plots that show the manifold tiling abilities of SDM in compar- ison to the other methods. Fig. 24 fits the UMAP projection on the weights of the pretrained on ImageNet32 models and then uses this projection for the CIFARIO data. Fig, 25 does the inverse where it fits a UMAP (MeInnes Healy 20188 projection to ConvMixer embedded CIFAR10 10 train- ing data and uses this projection for the trained model weights. Both Fig.s show that SDM learns to tile the data manifold most effectively with the full SDMLP being the only method to tile the manifold in both figures. This shows that upon pretraining, the neurons of SDM have learnt dif- ferentiate across the manifold of general image statistics, forming subnetworks that will be useful for continual learning
We emphasize that while there is manifold tiling and subnetwork formation, the ImageNet32 pre- training does not result in SDM already knowing the CIFARIO data or perfectly tiling the CIFARIO manifold. Fig. 27 shows UMAP plot with the projection fit to the SDMLP trained directly on CIFARIO pixels and projecting the same CIFAR1O data, note how much tighter the manifold tiling is here. Further evidence of manifold tiling is evident in this pixel based training where in Fig, 26 we take the weights ten random SDMLP neurons and reshape them into 3x32x32 dimensions to reveal the neurons have specialized to specific classes in the data. This is in stark contrast to the weights the ReLU neurons shown adjacently, SDM neurons specialize to not only specific image classes but specific examples within the class, analogous to the hypothetical "grandmother ron" (Gross) 2002; Quiroga et al. 2005). This figure shows the trained weights of randomly chosen neurons from an SDMLP trained on CIFARIO pixels and reshaped into their image dimensions. A bird, frog, and multiple deer and horses are visible.
Note that if we directly visualize these L normalized weights and our L normalized CIFARIO images, they all look black because the pixel values denoting RGB colors are between 0 and while our normalization makes them much smaller. We rescale our weights back into pixel values by first subtracting the minimum pixel value x min (x) and then multiplying * 1/(max (x)) that is standard practice.
(b)
Figure 26: Learned Neuron Weights for SDM (a) versus ReLU (b). We train our models directly on CIFARIO and visualize the weights of nine randomly chosen neurons by reshaping them into their image dimensions. SDM results in significantly more interpretable receptive fields for the neurons.
To fully emphasize that these neurons and their interpretable receptive fields are meaningful parts of the manifold, we take neuron that has a frog receptive field and show where is located on the manifold. We then scan the CIFAR10 training data and take the 3 images that maximally activate
The fact that the neurons unique and dispersed across the data manifold, when combined with the Top-K activation function, ensures that subnetworks of neurons of will be active for different input classes.
54

---

Published as a conference paper at ICLR 2023
SOM Training Data Weights
IPON OOCer
10
UMAP Dkm
(a)
(b)
Figure 27: Analyzing neuron receptive fields of SDM in relation to the data manifold. (a) UMAP plot fit on neuron weights trained on CIFARIO pixels (orange) with the CIFAR10 data (blue). The black dot in the top left shows the location of the neuron weights that give the frog image displayed as an inset. (b) The neuron weights are shown in the center, in the top row above are the three CIFARIO images that maximally activate this neuron showing that the neuron weights learn a superposition of similar frog images. In the bottom row are the three closest images determined by euclidian distance on the UMAP embedding. This embedding will be less precise but still shows that the manifold largely captures similar classes, aside from the horse.
this neuron, plotting them in the top row of Fig. 27 We also look at the UMAP embedded images that have the smallest euclidian distance the UMAP embedding of our neuron weights and show the closest three images. This UMAP apporach does contain a horse but this approach will be less precise due to the UMAP embedding mapping the 3072 dimensional images into 2 dimensions.
We use our neuron UMAP embeddings to visualize the number of times that each neuron is activated Fig. 28 This uses the same overall activation values across continual learning presented in the histograms of Fig. 22 with yellow indicating the most activations and purple the fewest.
Finally, we show the data manifold tiling of SDM in Fig. where assign each CIFARIO class a different color in the left plot and then show the positions of each neuron assigning the color that activates this neuron the most.
55

---

Published as a conference paper at ICLR 2023
SDM Number of Times each Neuron Activated
Dinn-LinarDeey/Mesk Number Times each Neuron Activated
UPA6
(b)
SDM Norm Number of Times each Activated
Top K Number of Times each Activated
URMO Dim
10
UMAP Dim
UNAP Dim
(c)
(d)
Number of Aclivated
U1000 Dim
10
UMAP Dim
Figure 28: Neurons colored by the number of times they have been activated. This plot makes it clear how SDM with no L norm (c) only activates a few neurons do all of its learning, resulting in catastrophic forgetting. Meanwhile, ReLU in (e) activates almost all of its neurons. All models that use Top-K: SDM (a), SDM with Top-K masking (b) and Top-K (d) activate subsets of neurons making it harder to distinguish their learning dynamics without the additional analyses presented in this section.
56

---

Published as a conference paper at ICLR 2023
Training Manifold by
Activating
UN6 Dlm 2
UMAP
10
12
10
UMAP Dim
UMAP Dim
(a)
(b)
Figure 29: SDM neurons tile the regions of the manifold that they most activated by. We use the same UMAP plot as in Fig. 25 but color the data by its class in (a) and the neurons by the class that activates each the most (b).
57


Reviews:
[{"review_id": "tN_ivyjg7i", "reviewer": "Reviewer_HvB9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with continual learning, and it is interested in a class of biologically plausible models to overcome catastrophic forgetting. Specifically, the authors focus on the Sparse Distributed Memory (SDM) model, a long standing memory model in computer science. The paper highlights a parallelism between SDMs and a single-hidden-layer Multi Layer Perceptron (MLP) and introduce some tweaks to the latter for empowering continual learning capabilities. Specifically, the authors employ in the hidden-layer a top-K activation function, resembling the sparse read operation in SDMs. Since its naive application introduces the problem of dead neurons, an annhealing strategy is introduced for reducing k (the number of active neurons) throughout the optimization procedure. The authors further notice how momentum-based optimizers are problematic in continual learning tasks, and simply advice not to use them. Experimental results are presented on Split-CIFAR-10 against a number of baselines.", "summary_of_the_review": "In my opinion the novelty of the work is limited for publication at ICLR. The authors reach remarkable performances on a custom task starting from pretrained embeddings, and I am lukewarm about how the model would perform in end-to-end continual learning (i.e. when representations are trained continually as well). The motivation of the work is purely biological, and there is limited discussion about why those ideas should work in practice.", "strength_and_weaknesses": "- +The paper overall well written, one read is sufficient to the reader to grasp the main ideas and contributions.\n- +The authors remarkably place the proposed model within the literature, with a lot of references to both neuroscience and machine learning prior works.\n- +The authors present a huge appendix that contains a lot of material useful for a deeper understanding of their research.\n- -The motivation that the authors highlight for the employment of SDMs is only biological plausibility. Although I acknowledge that as a good trait, the text does not convey the reason why, technically, the proposed model avoids catastrophic forgetting. My personal interpretation is that, given that the Top-K activation functions create sparse activation patterns, the corresponding back-propagated gradient is also sparse, and less prone to overwrite important knowledge learned during prior tasks. The authors should provide more intuitive explanations about why the proposed model is less prone to forgetting in technical terms, beyond biologically-inspired considerations.\n- -The paper describes how both read and write operations are performed in a SDM. However, the parallelism between SDM and a one-hidden-layer MLP is drawn only in the case of read operations (Eq. 3 and Eq. 2). How does the writing mechanism intervene in the authors model? Is the gradient-based learning of the MLP parameters considered the writing algorithm? If so, gradient-based updates do not conform with the writing mechanism for SDMs (Eq. 1). Can we still consider the model a SDM in that case? And if so, why?\n- -The main experiment on CIFAR10 presents encouraging results, where the proposed SDM trained incrementally learns better than other models that specifically intervene to reduce catastrophic forgetting. However, the setting is quite arbitrary, as the authors use pretrained imagenet embeddings and an SDM on top of that. This raises two concerns. i) given that imagenet embeddings already separate CIFAR10 classes (as the cifar10 class space is a subset of the imagenet class space), is this an interesting learning setting? ii) are all compared methods in Tab. 1 using the same protocol? In my opinion, the case where *both* the embeddings and the classifiers need to be learned is the most interesting scenario for studying continual learning problems. Can the proposed models be extended to arbitrary backbones (e.g. ResNets), where every pair of consecutive convolutional layers can be considered as a SDM?\n- -Table 1 would benefit a oracle where all classes are trained jointly, to put the results into perspective.\n- -In Table 2, different optimizers showcase very different performances on the task. Are all baselines in Tab. 1 comparable, meaning that they all use SGD without momentum as an optimizer?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W9Goxz6ry5", "reviewer": "Reviewer_1XFe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper interprets Spare Distributed Memory (SDM) as a continual learner. Analogous to a one-hidden-layer MLP, the SDM is modified to support continual learning. There are several training tricks, such as the choice of optimizer and the training procedure. The most notable change is to replace the binarization activation in SDM with the top-k function, where k is annealed over training time. The method shows improved performance in the class-incremental setting with  CIFAR10 dataset.\n", "summary_of_the_review": "The paper provides an interesting insight into SDM as a continual learner. The method, however, is over-complicated with many components and tuning efforts. The experimental result is weak. ", "strength_and_weaknesses": "### Strength\n- The paper gives a new insight: SDM can support continual learning, mainly based on its sparsity nature. \n- The paper has done a non-trivial task to make SDM work for actual data, which involves many tips and tricks in training SDM. \n### Weakness\n- The presentation is hard to follow. The main text lacks details to understand the method. The paper is not self-contained and often requires references to Appendix, which is only optional for the reviewing process.\n- Too many modifications to SDM make the final method no more SDM, which depreciates the paper's central message. Also, the amount of tuning hyperparameters and calibrating training procedures is enormous, making it hard to apply to a different CL data or task. \n- The experiment is limited. The proposed method (combined with EWC) is only better than FlyModel in CIFAR10, and underperforms in CIFAR100 and MNIST\n\n### Detailed comments and questions:\n- Eq. 3, how do you write to $X_v$? If you use SDM's write, what are the $P_a$ and $P_v$ in the case of CIFAR10 data?\n- The paper claims to fix the issue of random $X_a$ with top-K activation. However, Eq. 4 only presents the activation function, which affects the output $a$. How does it help $X_a$ model real-world data?\n- Additional modification: (v) How do you use backpropagation in SDM? (to update $X_v$, $X_a$?)  \n- Please consider adding an algorithm to the main text to clarify how your proposed components work together.\n- Please consider more datasets to validate the performance of your method: CelebA, FahsionMNIST, .... Given the current result; it is hard to say your method is more effective than FlyModel\n- Why don't you make stronger baselines by combining methods just as you did with SDM? For example, FlyModel+EWC or use more baselines such as EWC, SI, HAT\n- Check the section numbering. It should start from 1. Introduction ...", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GGE5bKk8eTn", "reviewer": "Reviewer_2ptq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors construct an architecture based on sparse-distributed memory (SDM) and multi-layer perceptrons and show that this architecture is naturally robust to catastrophic forgetting. This architecture uses sparse activations of neurons (top-K) with a couple of tricks to improve trainability. One one continual learning task (CIFAR-10), the authors show that this produces very good performance, esp. when combined with EWC.", "summary_of_the_review": "Overall, it is a well-written high-quality paper with very interesting and relevant results for continual learning. The paper's biggest weakness is having results (albeit detailed) on only one benchmark. The paper needs comparisons on a more diverse set of benchmarks, and is the primary reason for my rating.", "strength_and_weaknesses": "## Strengths\n\n- The authors clearly demonstrate that SDM in the specific form they use is suited for continual learning, with good ablation experiments\n- Detailed experiments on the one benchmark that is chosen.\n- The authors contribute a few tricks for training SDM-based architectures that are novel.\n- Methods are clearly explained.\n\n## Weaknesses\n\n- Evaluation done only on one benchmark.\n- Not clear if biological plausibility is a goal, and if so, how it is achieved.\n- Differences from (Shen et al. 2021) not clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CwW7j47p1tA", "reviewer": "Reviewer_krw5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a number of modifications to the standard Multi-Layered Perceptron (MLP) to avoid catastrophic forgetting in continuous learning tasks. The proposed modifications are motivated as biologically plausible and include the use of sparse distributed memory, a Top-K activation function, no bias terms, and L2 normalization and non-negativity constraints on weights and data. These features are studied in isolation and combined with other approaches, like Elastic Weight Consolidation (EWC). In addition, the paper proposes two training techniques: i) an implementation inspired on the \"GABA Switch\" to avoid \"dead neurons\", showing this is equivalent to annealing number of active neurons (K in the Top-K activation function) when the non-negativity constraint is imposed; and ii) using an optimizer without momentum to avoid updating neurons that are not in the Top-K set and another potential exploding gradients issue. The authors consider a three stages during training: i) first, pretraining a ConvMixer on ImageNet; ii) pretraining the SDM using the pretrained embedding; iii) continuous learning of the pretrained SDM module with the pretrained image embedding. Experiments show the proposed modifications to the MLP, combined with EWC, provide SOTA results for CIFAR 10 in the challenging class incremental setting, and achieves second better performance for CIFAR 100 and Split MNIST. \n", "summary_of_the_review": "This is a solid work, well written, full of insights, with promising results, and very relevant. I think it will foster future research on both SDM and continuous learning, and has the potential to help the representation learning community to be more aware of the potential of using neuroscience to improve standard deep learning architectures.\n", "strength_and_weaknesses": "# Strengths\nThe paper aims to provide a biologically plausible extension of MLP that naturally results in continuous learning capabilities. This is an important and usually overlooked research area. \n\nThe connection between SDM, MLP and Transformer is very interesting and it is developed in further detail.\n\nThe numerical results are very promising. Although the results are not SOTA for CIFAR100, they are still competitive and outperform other baselines. \n\nMore importantly, the extensive but practical explanations behind their design decisions, the insightful ablation studies, and the suggested directions for future work, will likely have an impact in the community and will foster future research.\n\n# Weaknesses\nSimulation results are promising. However, it would be reassuring to see a common trend with other data modalities, like text. For example, the authors could use a pretrained modern Transformer architecture (e.g., DistilBERT) and evaluate continuous learning while fine tuning to different tasks.\n\nThe motivation of a single layer MLP is not very clear. It would be interesting to see whether the results will hold for a multilayer MLP.\n\nThe explanation of the three stages training regime in the main text is not very clear (probably too concise due to lack of space). The authors could clearly enumerate the three steps, explaining a ConvMixer is used for pretraining from the beginning, and making clear what SDM Module refers to their SDM-inspired MLP variant (maybe coining their model something like \"SDM-MLP\" could prevent the risk of a reader thinking that SDM and MLP are two different models).\n\nThe ablation study on training directly on image pixels would be more relevant if they were done by tuning the loss coefficient and the $\\beta$ parameter for training directly on pixels.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster