Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning
Authors: Viet Anh Khoa Tran
Emre Neftci
Willem A.M. Wybo
Venue: NeurIPS 2025 poster ()
Keywords: cortical learning
top-down modulations
continual learning
self-supervised learning
semi-supervised learning
Primary area: neuroscience_and_cognitive_science
TL;DR: Predictive coding principles proceeding over modulated representations continually integrate top-down information from sparse labels in unsupervised representation learning.

Abstract: Biological brains learn continually from a stream of unlabeled data, while integrating specialized information from sparsely labeled examples without compromising their ability to generalize.
Meanwhile, machine learning methods are susceptible to catastrophic forgetting in this natural learning setting, as supervised specialist fine-tuning degrades performance on the original task.
We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex, using predictive coding principles to integrate top-down information continually and without supervision.
We follow the idea that these principles build a view-invariant representation space, and that this can be implemented using a contrastive loss.
Then, whenever labeled samples of a new class occur, new affine modulations are learned that improve separation of the new class from all others, without affecting feedforward weights. 
By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counterparts. 
This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it.
Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few as 1% of available labels.
Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.

Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning
Viet Anh Khoa Tran
Emre Neftci
Peter Grünberg Institute
Peter Grünberg Institute
Forschungszenttrum Jülich & RWTH Aachen v. Aaranffz-juelich..
Forschungszentrum Jülich & RWTH Aachen
.nnftciifz-juulll de
Willem A. M. Wybo Peter Grünberg Institute
Forschungszentrum Jülich w. sybo@fz-jue juelich, ch. de
Abstract
Biological brains learn continually from stream of unlabeled data, while integrat- ing specialized information from sparsely labeled examples without compromising their ability generalize. Meanwhile, machine learning methods are susceptible catastrophic forgetting in this natural learning setting, as supervised specialist fine- tuning degrades performance on the original task. We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical ma- chinery in the neocortex, using predictive coding principles integrate top-down information continually and without supervision. We follow the idea that these principles build a view-invariant representation space, and that this can be im- plemented using contrastive loss. Then, whenever labeled samples of new class occur, new affine modulations are learned that improve separation the new class from all others, without affecting feedforward weights. By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counter- parts. This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it. Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few 1% of available labels. Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.
Introduction
Input data streams encountered by animals or humans during development differ markedly from those commonly used in machine learning. In contemporary machine learning (e.g. foundation models), data streams typically consist of unlabeled data, augmented with some degree of supervised fine- tuning in the final training stages [1]. Such an approach is difficult to translate to the continual learning setting encountered in natural data streams, the naive introduction of fine-tuning stages often leads to catastrophic forgetting [2-4]. Meanwhile, animals and humans receive mostly unsupervised inputs, interspersed with sparse supervised data, which could, for instance, be provided through external teacher (e.g. a parent telling their child that the object is called an 'apple"). Compared to unsupervised data streams, such sparse supervisory episodes are infrequent. Therefore, the learning dilemma that
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

arises is how continual learning algorithms can benefit from sparse supervisory episodes without negatively affecting representations learned in an unsupervised manner.
Here, we draw inspiration from the circuitry in biological brains to solve this learning dilemma. We leverage the fact that cortical neurons
broadly speaking be subdivided into proximal, perisomatic zone, receiving feedforward inputs [5-7], and a distal, apical region, receiving top- down modulatory inputs [6-11 (Figure I, left). Through their physical separation, these zones are functionally distinct [12], and implement different plasticity principles [13, 14]. Learning of the perisomatic, feedforward connections is believed follow form of predictive coding [15, 16] that is the biological analogue of self-supervise learning such as VICReg [17] or CPC [18]. At the same time, top-down modulations to distal dendrites provide contextual, modulatory signal to the feedforward network [19-26]. We hypothesize that this signal is learned during the supervised learning episodes. In machine learning, this concept has been explored in the context of parameter- efficient fine-tuning by training task-specific scaling and/or shifting terms [27-32]. This provides a straightforward solution to continual learning problems for which the task identity is known during both training and evaluation (i.e. task-incremental learning [33]), modulations for new tasks can be learned without affecting core feedforward weights [34-36]. However, when the task identity is not known at evaluation (i.e. class-incremental learning), the modulated representations for each task need to be consolidated in a shared representation space.
To achieve class-incrementi incremental learning, we hypothesize, based on the spatial and functional segregation of distal dendrites, that the top-down signal not affected by the predictive plasticity of feedforward weights in the perisomatic region. As such, it leaves permanent imprint on the network that, through occasional reactivation, integrates the new percept the neural representation space, while also providing form of functional regularization that limits forgetting. We demonstrate that these effects are achieved by standard predictive coding principles that proceed over modulated representations. In our task-modulated contrastive learning (TMCL) algorithm, we use the currently available labeled examples for each new class to learn modulations that orthogonalize their representations from all others. These modulations are then frozen and applied to the network during the feedforward weight learning, using only currently available unlabeled samples, and effectively consolidate the task-specific knowledge encoded in the modulations into the feedforward weights (Figure I, middle). This departs from the conventional pretraining-fnetuninng paradigm, where naive reintegration of specialized models into the general one causes catastrophic forgetting [1, 4] (Figure 1, right).
We evaluate the performance of TMCL on the standard class-incrmmentl CIFAR- 100 benchmark, outperforming state-of-the-art purely unsupervised, purely supervised, and hybrid approaches in label-scarce scenarios. Furthermore, we evaluate its transfer learning capabilities across diverse set of downstream tasks, demonstrating its effectiveness in learning generalizable representations that extend beyond adaptation to CIFAR-100 Finally, we show that our method dynamically navigates dilemma through adaptation of the consolidation term.
the
2 Related Work
Biological representation learning. Several authors have explored the idea that the cortex learns in a self-supervised manner [15, 16, 37-40]. Although complementary approaches based on adversarial samples have also been proposed [37], most theories focus on some form of predictive coding, where the cortex learns to predict the next inputs given the current neural representation. Mikulasch et al. [38, 39] take a classic view on this, where loss function the next layer reconstructs the input, while Kermani Nejad et al. [40] theorize that the architecture of the cortical microcircuit is well-suited for predictive coding. Finally, Illing al. [15] and Halvagal and Zenke [16] propose local plasticity rules based on, respectively, CPC [18] and VICReg [17] that, as they argue, in a natural setting could proceed by comparing neural representations at subsequent time steps. We extend this idea with an explanation of how top-down modulations could be incorporated into the learning process.
Learning with modulations. The expressivity of learning modulations was initially demonstrated by Perez et al. [27] to solve visual reasoning problems. Frankle et al. [28] subsequently showed that a surprisingly high performance can be achieved with ResNets [41] while just training BatchNorm layers, which - if performed per-task - is equivalent to learning affine modulations. Finally, it was shown simultancously in language and vision that fine-tuning through modulations in transformer

---

Cortical Learning
Task-Modulated Contrastive Learning
Machine Learning (transfer learning) Pretraining (unsupervised)
Top down thigh-level information)
Consolidation (unsupervised)
"human" "tree"
A grod)
Crt
CCL
Wgper?
AC (MMoro))
nurmany
Tree?
SeWW JEEE
/t (X)
Feedforward (precicttive coding)
A
t-
Wase
d
top-down modulation
learning (sparsely supervised)
afternating
Fine-tuning (densell supervised)
Cortina
X
XX
LiB
X-X
feedforward
drive
human, tree, car?
ft W.
SGJW,
agg
forward pass
forward pass stop-grad
trained parameter
Figure 1: Biologically inspired consolidation of high-level modulations into feedforward weights. Cortical learning (left) is characterized by the interplay between top-down (orange) and feedforward (blue) processing, where top-down connections impart high-level information on the feedforward sensory processing pathway (top). The feedforward pathway, on the other hand, learns to predict neural representations of future inputs (predictive coding). Notably, top-down and feedforward information arrives at spatially segregated loci on sensory neurons (bottom), suggesting distinet roles in shaping the neuronal input-output relation (cf. [19]) well distinct plasticity processes governing weight changes. Translating this view to machine learning algorithm (middle), we (i) train modulations to implement high-level object identification tasks as the analogue of top-down inputs (bottom, solid arrows, but not dashed ones, indicate that gradients backpropagate in the opposite direction, and underlined parameters are trained), while we (ii) train for view invariance over modulated representations and thus also for modulation invariance the analogue of predictive coding (top). As a consequence, high-level information continually permeates into the sensory processing pathway, which can be contrasted with the traditional machine learning (right) approach of unsupervised pretraining for view invariance (top) followed by supervised fine-tuning (bottom). In this case, it is unclear how high-level information be incorporated into the sensory processing pathway to improve subsequent learning.
models particularly powerful, as it reaches the same performance as using all parameters [29, 30, 32).
Modulations are attractive way implement task-incremental learning, as task-specific modu- lations can be learned for each new task without affecting feedforward weights. Masse et al. [34] propose gating random subsets neurons, whereas Iyer al. [36] provide biological interpretation. Fine-tuning through modulations [30, 32] can also be considered as a form of continual learning, as can be applied sequence to any new dataset.
Continual representation learning. Traditionally, continual learning has focused on purely super- vised methods [42-67]. These methods can be categorized into replay-based approaches [42-50], regularization-bal based approaches [50-62, 67] and approaches introducing new parameters [63-66]. TMCL can be considered regularization-base approach, but it also introduces new parameters. However, these parameters not used during inference. Recently, purely self-supervised con-

---

tinual learning algorithms have been proposed [68-72], where ofate-of-the-arr algorithms [69, 72] predict past representations from a stored model copy without exemplar replay. Very recently, semi- supervised continual learning approaches have emerged [73-76], which consolidate by distilling from expert models [76, 77] or for which the labels are only used for readout learning [74].
We highlight SIESTA [49], CLS-ER [78] and DualNet [73], which are inspired by the complementary learning systems theory (CLS) [79, 80], the idea that learning occurs at fast (task-learning) and slow (consolidation) timeframes. However, all these methods interpret CLS to assume sample replay provided as episodic memory via the hippocampus. Instead, we suggest functional replay of task modulations ( ('How did we solve the task?"), and do not investigate methods with exemplar replay. Still, we point out similarities to the replay-baset DualNet, which introduces fast supervised network generating modulations on top of slow self-supervisee network. However, DualNet consolidates only as new labels arrive. Consolidation TMCL. requires no labels, instead exploiting previously learned task modulations.
3 Modulation-Invarttt Continual Representation Learning
We follow the idea from lyer et al. [36] that cortical networks learn to interpret novel information by learning new top-down modulations, and propose that consolidation of these modulations is crucial component of learning in biological brains. This motivates our medullodet contrastive learning (TMCL) algorithm the machine learning analogue of this consolidation, tackling continual representation learning. We consider the parameters of conventional machine learning models as lask-agnostic feedforward weights W. On top of these weights, we introduce per-task affine transformation parameters as task-specific modulations m,, the analogue of biological top-down modulations. We denote the modulated network as f(x W, m) with feedforward weights W and modulations m. while fxx W. 0) represents the unmodulated network (i.c. where the modulations are identity operations).
In TMCL, the overall objective is to arrive at an unmodulated representation space where all classes E C in the dataset 'D have compact representations clustered around mutually orthogonal class centers, i.e.
PL Ve, d' EC,
with y = Ex [f Wx 0)). where ((() is the set of samples from class c. Because we assume a continual learning setting, where do not have all class samples for (1) directly. Rather, we achieve this by breaking the optimisation procedure down into two distinct learning objectives. The first objective (Figure 2, bottom left) is orthogonalize any given class c from the others in a modulated representation space, i.e. we learn a modulation m so that the class center of class becomes orthogonal to all other classes in the modulated space:
disposal, we do not optimize
Im ( : :'' E (cH),
(2)
where 7m denotes the representation of the class center under modulation m, i.e. T Exex(e) (f()"(,)) The second objective (Figure 2, bottom right) is entirely unsupervised and trains network weights to become modulation-immariann so that
7 - "n Vc' E C.
It can be seen that representation space that satisfies both (2) and (3), also satisfies (1).
In our continual setting, which we adapt from Fini et al. [69], training is partitioned into 8 E 1,..., sessions, so that (1) can only be achieved approximately. In each session s, we only observe unlabeled samples E D C 'D belonging to the session-speciffs partition of classes ((o) C C. Additionally,
fraction of labeled samples (Ir y) E DDa DD()) is made available (3). As consequence, during each session (Figure 2), we learn objective (2) restricted to sup in first phase, and then learn objective (3) using unlabeled samples from D(.), We explain the implementation of both phases in detail below.
Learning modulations that orthogonalize new class representations. Whenever a new class label is observed, we learn class modulations on top of frozen feedforward weights to implement objective (2), which improves separation of the new class representations from all others currently

---

session
few new labels
Input stream unlabeled
TRR
"hale"
TRger
Orthogonalization
Consolidation
warn Lvinas
dineaileeee frone
scssiant 1 session
session 1
Supervised Modulation Learning
Contrastive Feedforward Learning present
explict one-ve-rest orthogenalization with present samples
implicit orthogonalization
"Eat" Aadion spoce
"uhiil PIdd JoSiC CTUCE (SIMeRT))
UnvecSaated
(uort)
Figure 2: Sparsely labeled class-incremental representation learning. We implement continual learning over mostly unlabeled data streams, where only few labeled samples are provided (top). To give intuition of our algorithm (bottom). consider that after successfully incorporating the data seen thus far, sufficiently collapsed neural representations exist for the already seen data classes after session - 1 (here dog, cat). For collapsed representation may not yet exist. We then learn a new set of modulations to collapse "whale" representations in the modulated representation space, orthogonalizing them from all other available labeled examples, thus obtaining an orthogonal subspace for everything that is non-whale. Then, occasional reactivation of the "whale" modulation in LcL draws unmodulated "whale" representations towards this collapsed representation (cf. Figure 1, middle), while drawing other samples to the orthogonal subspace, thus consolidating "whale" into the unmodulated representation space.
data class in session t (e.g. whale), such
available. We assume that explicit class labels for these other classes conventional machine learning approach of all-vs-all classification is not applicable. Instead, we learn one-vs-rest class modulations, only using currently available samples as negatives (Figure 2, bottom left). Note that if negative samples were to collapse to a single representation, (2) and (3) could hold simultaneously and therefore (1) could not be achieved either. For this reason, we apply variation of the orthogonal projection loss (OPL) [81] instead of binary cross-entropy (BCE) [82]. We define sm (u, v) sim(f (u/ fv m)) as the cosine similarity between samples ? and v under modulation m, i.e. sim(u, (u. v) = ||ж!||!! Then, given batch ((e) of c-class examples
available, therefore, the
and batch (-c) of non-c examples, we define
(c) CO
Σ
9m (p.p') collapse
+
Σ pEX(c) nEX(
Ism (p. n)
"orthogonalization
(e) is then found as min, [(e) The second term draws the cosine similarities between class and non-class representations zero, leading to orthogonalization of class representations from all others, therefore approximating objective (2).
Consolidation modulations into a view- and noodulationinnariian representation space. To implement objective (3), we co-opt self-supervised contrastive learning, which is considered a biolog- ical analogue of predictive learning principles of the feedforward connections [15, 16]. Contrastive learning objectives train for view-invariance (VI), as they attract representations of views of the same source sample under different view augmentations to each other, while repelling representations of forming representations colloquially
other samples [17. 18, 83-92]. These view augmentations referred to as 'positives", are sampled randomly from set of augmentations A (i.e. a ~ A), which includes combinations of e.g. random crops, color jitter and horizontal flips. We note that modifying the set of positives, while using the same contrastive learning objective, results different invariances being learned. We generalize the contrastive loss LCu(IzI, Zk generic learning rule
5

---

operating on set of K positives Z1, 21...., Zx. Then, we formalize the view-invariant loss as
VI = faL (a(( W. = 1...., K, Og ~ A}),
(5)
where Pvv = hvz o and hvr(-) is an MLP used exclusively for the view-invariant objective.
To consolidate the orthogonalizing modulations into the unmodulated representation space. we propose employ differently modulated views of the source sample positives, hence constructing modulation-invarian representation space (Figure 1, top center). Concretely, we uniformly sample m2..... my (M from the set of trained modulations at session s. We formalize the modulation- invariant loss, which implements objective (3),
MI := LcL (P W. 0))}
(6)
u of ( mx)) h: == K, ar A, m. M((e))
where sg(-) denotes a stop-gradient, şPI = hyı o f. and pmr pmr(.), hwr (-) are MLPs used exclusively for the modulation-invariaa consolidated in the unmodulated representation space. During the consolidation phase of TMCL, the combination of VI and MI is optimized jointly.
objective. Therefore, compact representation of each new class is
Mapping prior work to the canonical contrastive loss and comparing to TMCL. Notably, in supervised contrastive learning (SupCon), samples of the same class are used positives [93], thus implementing invariance to features not predictive of class identity. SupCon is straightforward method to additionally leverage the few available labels in semi-supervised continual learning setups. However, in contrast to our MI objective, SupCon only separates classes for which labels are available in the current session, while MI implicitly separates current samples from past class centers.
State-of- the-art art unsupervised continual representation learning algorithms such as CaSSLe [69] and PNR [72] train the network representations to be invariant to the model state (state invariance or SI), by using as positives one representation obtained from the current network state and one representation obtained from stored, past network state. In TMCL, because the modulations for any given class are nos updated after the initial learning, they effectively constitute an imprint of neural activities that separates that class from all others. Therefore, training the unmodulated representations to maintain similarity with this imprint also stabilizes continual learning, and can be understood as form of SI that does not require storing the full network state twice (cf. Table A1, right).
Experiments
Experimental protocol. We adopt standard class-incremental continual learning protocol on both CIFAR-100 and ImageNet- 100, dividing the dataset into five sessions, each containing 20 disjoint classes. We additionally introduce supervised cross- cross-entropy (CE) baseline with a projection head [94], which has been reported to outperform self-supervised methods. For each session, we train the model for 100 orthogonalization epochs, where train modulations for the new classes, and 200 consolidation epochs. We further start the first session with a pretraining phase of 250 epochs, where the feedforward weights are updated via VI, or analogously with SupCon and CE for the respective supervised methods. For all loss terms except CaSSLe and PNR, we use K = positives. The backbone is modified Con ViT architecture [95] as introduced in DyTox [65]. We emphasize that this architecture is equivalent to ResNet-18 18 in both memory and compute (Table A1, b). We evaluate the representations via linear probing of the last four layers of suggested by Caron al. [88]. Further details are provided in supplementary materials A.
Semi-supervised continual representation learning. In Table 1, we present the all-vs-all linear readout accuracies on the CIFAR- 100 and ImageNet-100 datasets after class-incremental learning. As previously reported [94], continual supervised learning outperforms self- self-supervised learning if all labels are observable, while purely supervised methods significantly degrade with only 10% of labels or fewer. Clearly, combination of both supervision and self-supervision proves most performant in this setup, as VI + SupCon significantly outperforms the state-of-the-art unsupervised algorithm (VI + SI (PNR)). While state invariance is helpful, most of the improvement stems from the additional supervision (VI SupCon). In the fully labeled scenario. VI + MI improves

---

Table I: Semi-supervised continual representation learning. Final linear readout accuracy (all-vs- all) on inssemnncremttall ImageNet-100 100 and CIFAR-100 with 5 sessions (averaged over three and four seeds respectively, = denotes the standard deviation).
CIFAR-100/5
ImageNet-10005
Method
100%
10%
1%
100%
10%
1%
Supervised methods
SupCon
58.4
47.7.009
39.1
64.0.007
48.7:112
33.6
CE
60. 10.
50.7.00
41.6200
66.02a.4
10.1
35.2
Self-supervised methods
VI
59.3
59.7.00
integrating class labels (semi-supervised)
+ CE
60.6206
59.520a
59.8
64.5
61.4
60.3
+ -SupCon
62.2
59.62n4
59.3
66.6-2.4
61.4
60.2
MII (TMCL)
60.7
61.1 10.3
60.7
64.5
63.5
62.0
introducing state invariance (and class labels)
+ SI (PNR)
60.22202
59.6
+ CE
61.2=a3
60.3
60. +03
64.5
60.3
59, =9.4
+ SupCon
62.7 -0.2
60.7
60.1 +0.3
67.0
60.2
58.5
MI
60.9
60.7
60.9
63.8
62.7+to6
61.7=n4
Table Transfer learning. Final all-vs-all kNN accuracy on diverse downstream tasks after five incremental CIFAR-100 sessions (averaged over four seeds, + denotes the standard deviation).
Method
Aircraft
CIEAR-10
CUBirds
DTD
EuroSAT
GTSRB
STL-10
SVHN
VGGFlower
1% CIEAR lahels
SupCon
8,3
52.0
3.3
15.5 vas
64.1
38.5
44.9
46.6
18.9-2o
+ (CaSSLe)
11.6
51.9
3.4
16.3
66.4
38.3
44.8
46.3
21.9
VI
27.5
77.0g
10.0ga>
27 67.6
86.0ane
67.8
65.4gns
48. 3nn
58.5
SupCon
27.4
77.3
9.8
27.9
85
68.2a1a
64.9
49.8
58.4
MI (TMCL)
28.0
78.0
10.7
29.4.ss
87.1
68.2
66.3
49.0
61.7
(PNR)
28.5
78.3
11.1a
28.
87.0aaz
69.4
0..0a
49.
64.7ato
SupCon
29.1
78.3
10.3
28.2
87.0
70.2
66.8 to3
49.9
64.4-03
29.9
79.0
11.8
29.5
87.5
69.8
49.7
66.2
100% C.I.
CE [94]
29.0
78.4
10.0
28.5
83.4
64.0
66.2 i65
53.2
$8.2
VI
27.5
77.0
10.0.a3
27.6,ns
86.0ae
67.8
65.4
48.3
58.5
SupCon
28.1
79.1
104
28. 28.9
86 86.6211
68.62ns
66.8.s2
49.8
58.2
28.9
78.2
10.7
29.2
87.0.aa
71.0 c.4
66.7
50.7
62.8
VI SI, yet underperforms VI + SupCon. This changes move towards label-sparse scenarios, where MI consistently outperforms SupCon. Interestingly, MI not orthogonal to SI, as their combination results in the strongest performances in label-sparse scenarios.
Representational quality for transfer learning. In Table 2, we report k-nearest neighbors (kNN) performance of different tasks top of the CIFAR-100 continually pretrained models, without any further fine-tuning or training. First, on those models that only observe 1% of the training labels (or no labels at all, i.c. VI, VI SI), we demonstrate that modulation invariance improves representational quality beyond solely adapting to CIFAR- 100 classes, as performances across almost all probed datasets improve. Here, MI is not orthogonal SI, and the combination of VI+SI+MI results in the most transferable representations. Furthermore, amongst methods exploiting CIFAR-100 labels, MI outperforms SupCon, supporting the hypothesis that direct supervision results representations that are "greedily" invariant to features which are not relevant for the current task. Strikingly, this cannot be solely attributed to the sparse supervision setup; conducting the same study on models trained with 100% labels, we observe that MI outperforms SupCon, except for CIFAR-10 and STL-100, which are semantically similar to CIFAR-I00 Furthermore, semi-supervised + MI outperforms fully supervised CE most tasks, further underlining the lack of generalization of class-based invariance
7

---

((3)
((2)
(0) accuracy
60%
Table 4: Backwards and forwards transfer per- formances, using 1% of labels for SupCon and MI (averaged over four seeds, + denotes the stan- dard deviation).
50%
40%
8T
81
BT
1
4
5
1
3
5
2
4
C(4)
((3)
Methods
BT
FT
et) accuracy
60%
SupCon
-1..5+0.4
-2..7+0.4
50%
+I (CaSSLe)
-18.1
-27.4+0.4
40%
FT
87
-1.7
-6.5+0.4
1
2
3
5
1
2
3
5
MI
-0.3
-3.7
Session
+ SupCon
-1.9
15.
Figure Controlling the stability-plasticity
+ SI (CasSLe)
1.0=0.1
t0.4
trade-off. Accuracy CIFAR-10 test sam- ples associated with C(s), measuring forward and backward transfer (averaged over four seeds).
+ MI
1.3
-3.0
+ SupCon
1.0=0.1
-3.7
Color scale indicates the strength of MI.
learning. In contrast to the CIFAR- 100 results (Table 1), observe that more labels provided to MI, representational transfer quality improves for most tasks.
Robustness noisy labels. Erroneous labels are commonly encountered in natural learning environ- ments. While in that case, misclassification sam- ples would be expected, the quality of the repre- sentations should not deteriorate. However, intro- ducing label noise severely degrades performance in fully supervised methods (Table 3, red cells de- noting performance inferior to the self- supervupede baseline), and naive integration via CE also signif- icantly underperforms the self-supervised baseline. While VI + SupCon only degrades with as much as 90% label noise, TMCL. is robust label noise up 99% and does not underperform the self-supervie supervseee baseline. We hypothesize that this robustness arises because in TMCL, the erroncous labels do not di- rectly affect weight learning. Rather, they lead nonsensical modulations, which represent a noise contribution to which the contrastive learning machinery learns to become invariant.
Table 3; Label noise. Final linear readout accuracy on continual CIFAR-I00, replacing fraction of labels with random labels. (aver- aged over four seeds, + denotes the standard deviation).
Method
Label noise
30%
50%
90%
99%
Fully ipperiiee methods
54.4
8.2.11
6.4
SupCon
Self-supertoed bazellve
integrating (mnisy) labels
CE
59.2
59.2 2.
58, 3.1-
58. Jass
Supdoz
60.6
60.0
58.6.1
58.7 7..
(TMCL.
60.4 60.4
.0.3
The strength of MI controls the stability-plastiicl trade-off. We first investigate how different methods perform before and after observing task samples. Therefore, we introduce backwards and forward transfer metrics that measure the difference task performance pre- and post-session compared to model that is trained solely on that particular task using VI (definitions provided in the supplementary material). We observe that strongly improves both backwards and forward transfer compared to pure VI, while SupCon only provides mediocre improvements in forward transfer and even degrades backwards transfer (Table 4). Introducing a model state invariance term significantly enhances backward transfer, demonstrating positive knowledge transfer from previously learned tasks. Remarkably, the combination MI and SI on average achieves performance within percentage points of the task-specific baseline model, demonstrating significant forward transfer. To further investigate the effect of MI, we vary its strength AMI E [0, 1] in the combination VI + AwrMI. In Figure 3, for each session s, we observe the accuracy of test samples associated with classes from ((e) during the course of training. We observe that increasing Awz enhances both forward transfer (left of the gray region) and backward transfer (right of the gray region), albeit at the expense of current task performance (gray region).
8

---

Table 5: Ablations.
3.4
Pretraining
Consolidation
(a) Ablating the role of orthogonalizing modulations. Linear readout accuracy on continual CIFAR- 100 and ImageNet- 100 observing all labels (average, + denotes the standard deviation).
3.2
3.0
CDNV
Method
CIFAR-100
ImageNet-100
VI MI (TMCL)
60.72a4
63.5
untrained mods.
60.1
60.6
2.6
random mods.
59.7=as
60.9
OPL + BCE
59.5
63.8
2.4
w/o MI
59.3
59.7.aa
2
3
5
Session
(b) Ablating the role of architectural choices. Linear readout accuracy continual CIFAR-100 (averaged over four seeds, denotes the standard deviation).
(OPL, labels)
MI (rand. mod.)
MI (OPL, 10% labels)
MT (BCE)
MT (OPL, 100% labels)
Method
CIFAR-100
Figure 4: CDNV during consolidation. Mea- sured on the CIFAR-100 test split on runs observ- ing all labels (averaged over four seeds, shading indicates min, and max.)
VI MI (TMCL)
60.7 20.
w/o pred.
55. 1 10:1
w/o stop-grad
60.3
w/o pred. & stop-grad
60.3
Orthogonalized modulations improve class- separation. We hypothesized in Section that BCE is suboptimal, since negatives are collapsed towards a single vector. As we replace the orthogonal projection loss with binary cross-entropy, degraded performance on CIFAR- (Table 5b). On ImageNet-100 performance is similar, which we attribute to the fact that the BCE training does not fully converge
Table 6: Results on ResNet-18. Reporting
linear evaluation performances continual CIFAR-100 (averaged over four seeds, + de- notes the standard deviation).
observe
Labeled frac.
Method
100%
10%
antiparallel configuration: On CIFAR-100, the average minimum BCE loss over all classes is 0.005 while on ImageNet, it is 0.053 10.03. Interestingly, while untrained modulations (i.e. frozen after normal initialization) as well as random modulations (i.e. redrawn each time) degrade performance further, they still provide moderate improvement over pure VI. This suggests that predicting representations based on a slightly perturbed model state already provides a regulatory effect. We further measure the class-distance normalized variance (CDNV, appendix E) of representations [97], which is the ratio between intra-class variance and class-mean distances. A lower CDNV thus indicates a combination of higher intra-class collapse and/or higher distances between classes. We observe that modulation invariance improves class- separation the CDNV significantly decreases once training enters the consolidation stage, but only in combination with orthogonalizing modulations (Figure 4). Moreover, the effect is still clearly visible while using 1% labels.
VI
53,4
+ MI (TMCL)
58. 1 #0.1
56.7 +
+ SupCon
54.5 a0.2
54.6
+ SI (PNR)
59.9 a0.3
SI (CaSSLe)"
60.1 +:04
SII (PNR))
60.3
+ ER [61]
54.6
+ DER" [48]
55.3
+ LUMP* [96]
57.8
+ Less-Forget" [51]
56.4
+ +POD* [56]
55.9
CLS-ER [78]
56.7 =0.2
49.8
"'dapted from Cha et al. [72]
"adapted from Fini et al. [69]
Architectural ablations. The predictor network appears to be essential for MI, but only if a stop- gradient imposed upon the modulated branches (Table 5b). Removing the stop-gradient while keeping the modulations frozen results in a moderate performance degradation, regardless of the presence of predictor. Still, the nop-gradient& useful from both a computational perspective - reducing memory footprint and gradient computations and from biological perspective, as the

---

predictor with stop-gradient could be implemented by hypothesized cortical predictive coding pathway [40].
ResNet architecture. As we replace Vin with ResNet-18 (Table 6), we observe that SI significantly improves performance over standard VI, underlining the susceptibility of ResNets to forgetting compared ViT-based architectures [98]. Introducing MI also significantly improves performance over VI as well as other label-integrating methods (SupCon, CL.S-ER), but slightly underperforms SI. We hypothesize that the ability to directly modulate spatial relationships in ViTs lends more expressivity to the modulations in these models. Furthermore, the reliance of ResNet architectures on BatchNorms hinders stable training of different modulations, as it requires freezing the batch statistics. This is observable especially in the high standard deviation in accuracy of TMCL given 10% labels. As the underlying data statistics change with each session, the BatchNorms will adapt accordingly, and we hypothesize that this interferes with previously learned modulations. This is not the case for LayerNorm as used by transformer architectures which normalizes across features and does not depend on stored normalization statistics.
5 Limitations
Our work focuses on developing an understanding of algorithms that leverage the cortical circuitry, which is characterized by top-down and bottom-up information streams [99], to learn in naturalistic scenarios. Our analysis demonstrates this on a standard CIFAR- 100 class-incremental setup, but omits data- and requirements that scale with the number of classes. For 100 classes, this amounts to around 4. IM parameters, while CaSSLe and PNR store a copy of the network (10.7M, Table Alb). Still, alternatives such as pruning simple classes or, better yet, a network that generates such modulations. should be explored. Finally, we observe that representational quality on the trained dataset (CIFAR-100) only moderately improves as more labels are presented, underperforming SupCon in the fully labeled regime. Still, given the improved transfer performance of TMCLL, we hypothesize that this is indeed biologically plausible as the unmodulated network is not primed to solve CIFAR- 100 specifically, but rather driven towards generalizable representations. Top-down priming could be implemented via separate set of task-specific modulations, as has been explored in previous work [34-36].
TMCL relies on static modulations, resulting in memory
6 Discussion
In this work, have proposed novel, brain-inspired algorithm for continual learning. It has been proposed that modulations provide a powerful framework for task-incremental learning, as they allow general feature detector learn new tasks by adapting modulations only [34, 36). We extend this framework to inssemnncrmmetal learning, and show that modulations can be consolidated into a shared representation space, sharpening percepts from data classes observed asynchronously. This consolidation co-opts the general machinery for view invariance learning, which in the brain is thought to be available anyway for predictive coding [15, 16]. Furthermore, there appear to be measurable parallels between our algorithm and cortical learning, as large-scale brain imaging indicates that representations of new stimuli orthogonalize from all others throughout learning, and unsupervised pretraining affects task learning [100]. Elucidating the drivers of this orthogonalization will provide further insight into how the brain leverages its biophysical machinery to achieve continual learning.
As we have shown that training for modulation invariance imparts task-specific information on the unmodulated network, one tantalizing possibility is that the effective learning objectives for networks, or parts of networks, could be tuned by incorporating sets of modulations that solve specific tasks. In mixed language-vision approach [27], this could afford fine-grained control over the eventual representation learning beyond what is currently possible with e.g. multi-task datasets [101]. In neurobiology, this represents a new view on intra-cortical and thalamocortical interactions. As cortical regions are targeted by modulations originating from distinct sets of brain areas with specific roles [102], the precise configuration of modulatory afferents could provide an understanding of the effective area-specific learning objectives. In turn, as these connectivity patterns are driven by genetically determined cues, this theory may afford insight into the distinct roles of genetics and plasticity in developing functional brains [103].
10

---

Acknowledgements
This work funded by the Federal Ministry of Education and Research (BMBF) under grant no. 011S22094E WEST-AI as well as by Helmholtz Association's project-oriented funding pro- gramme (PoF 2, Topic 3). Furthermore, the authors gratefully acknowledge computing time on the supercomputers JURECA [104] at Forschungszentrum Jülich under grant no. jinm60. The authors also gratefully acknowledge the Gauss Centre for Supercomputing e. (www.gaus www.gauss-centre.co funding this project by providing computing time through the John von Neumann Institute for Com- puting (NIC) on the GCS Supercomputer JUWELS [105] at Jülich Supercomputing Centre (JSC). The authors also extend their gratitude to Sven Krauße for providing thoughtful comments on the manuscript.
for
References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael $ Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, and others. On the opportunities and risks foundation models. arXiv preprint arxiv:2108.0777
2021.
[2] Sylvestre- Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Isahelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Gamett, editors, Advances Addances neural information processing systems 30: Annual conference on neural information processing systems 2017, december 4-9, 2017, long beach, CA, USA, pages 506-516, 2017. URL https:// ppooeeddigg/edggss neurips cc/ cc/paper/ 2017 htthaahhhhh/bbba// e7b24b1 122444fddeeeeeaa998oooo..o scienee bibliography, https://dblp.org tex. timestamp::: Tue, 07 May 2024 20:04:34 +0200.
Abstract .ttll. tex. bixbbusouecee: dblp computer
[3] German Parisi, Ronald Kemker, Jose Part, Christopher Kanan. Stefan Wermter. Continual lifelong lcarning with neural networks: A review, Neural Networks, 113:54-71, 2019, ISSN 18792782.
doi: 10.
0..2.. Publisher: Elsevier Ltd.
[4] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song. Hannanch Hajishirzi, Simon Korbblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. net/
Advances Neural Information Processing Systems, 2022. URL
[5] Rodney J. Douglas and Kevan A.C. Martin. Neuronal Circuits of the Neocortex. Annual Review of Neuroscience, 27(1):411-455) July 2004. ISSN 0147-006X, 1545-4126. doi: 10. 1..11///muurev.eeuuro 27. 0772203.141522 URL https: / http::://wwww../// annualreviews org/ /oo//0..111111//// 1146/annurev annurev. .wwroo.erro.. 070203 144152.
[6] Satoshi Manita, Takayuki Suzuki, Chihiro Homma, Takashi Matsumoto, Maya Odagawa, Kazuyuki Yamada, Keisuke Ota. Chie Matsubara, Ayumu Inutsuka, Masaaki Sato, Masamichi Ohkura, Akihiro Yamanaka, Yuchio Yanagawa, Junichi Nakai, Yasunori Hayashi, Matthew E. Larkum, and Masanori Murayama. A Top-Down Cortical Circuit for Accurate Sensory Perception. Neuron, 86(5):1304 -116,, 2015. ISSN 10974199. doi: 10. 10.1016/j.nnureee new.on. 201..0015000 1.005.00.66. URL http: //tp://xx..ooo org/10. org/1..101//] neuron. 2015 00.1..05.000 006. Publisher: Elsevier Inc.
[7] Mathicu Lafoureade, Maric-Sophic H. Van Der Goes, Dimitra Vardalaki, Norma Brown, Jakob Voigts, Dae Hee Yun, Minyoung E. Kim, Tacyun Ku, and Mark T. Harnett. Differential dendritic integration of long-range inputs in association cortex via subcellular changes in synaptic AMPA to No-M receptor ratio. Neuron, 10 www.wwer...e.................... http:////ee.reeee/////////////00
May 2022. ISSN 08966273, doi:
URL httpp:://iikkiii
[8] Morgane M Roth, Johannes Dahmen, Dylan R Muir, Fabia Imhof, Francisco Martini, and Sonja Hofer. Thalamic nuclei convey diverse contextual information to layer of visual cortex. Nature Neurscience,, 1922)299-3377 February 2016. ISSN 1097-6256, 1546-1726. doi: 10.1 URL httt::/www.nateeee
www.rattre..... com/ httee/atticceeeeeeeeeeeeeeeeeeee nn17997.
[9] Naoya Takahashi, Thomas G Oertner, Peter Hegemann, and Matthew Larkum. Active cortical dendrites modulate perception. Science, aah6066.
2016. ISSN 0036-8075. doi: 10. 16.112//cience.
[10] Guy Doron, Jiyun N. Shin, Naoya Takahashi, Moritz Drüke, Christina Bocklisch, Salina Skenderi, Lisa de Mont, Maria Toumazou, Julia Ledderose, Michael Brecht, Richard Naud, and Matthew Larkum. Perichinal input to neocortical layer controls learning. Science, 370K6523) 136. December 2020. doi: 10. 101116/cience..arz2 a323136. URL https: //www/wwwww//// wwwpwrscccccee..eeec.eee orw//oii//itoooooooooooooo dt////o1111.001 1126/ science. aaz3136. Publisher: American Association for the Advancement of Science.

---

[11] Benjamin Schuman, Schlomo Dellal, Alvar Prönneke, Robert Machold, and Bernardo Rudy. Neocortical Layer I: An Elegant Solution to Top-Down and Bottom-Up Integration. Annual Review of Neuroscience,
44(March):221-2555 March 2021. doi: 10. 446/anmurev-neurr 100520-012117. URL httpp::/wwtt annualreviess.
www.ooio////oooooooooooooooooooo htt///////p/////// /10. wttp///nnnueeveeeeeeee neuro 00002000022111
[12] Matthew Larkum, Thomas Nevian, Maya Sandler, Alon Polsky, and Jackic Schiller. Synap- tic Integration Tuft Dendrites of Layer 5 Pyramidal Neurons: A New Unifying Principle. Sei- ence, 325(5941) August 2009. ISSN 0036-8075. doi: 10.1126/science. 1171955. URL
http://wuz scwencemag........ org/cgi/do//10. 11tp:/11/11////eeeeeeeeee htt.://ecceee..11..... 1171958.
[13] Bjöm M. Kampa, Johannes J. Letzkus, and Greg J. Stuart. Requirement of dendritic calcium spikes for induction of spike-timing teeeeeiiinngdeeeedeentt synaptic plasticity. The Journal of Physiology. 574(1):283-
290, July 2006. ISSN 0022-3753 3751, 1469-7793. doi: 10.1113/jphysiol.2000 11062.. URL https:
/ h/physoc.onlinge inw.neibbrrr........mmm......ii. wiley- com/doi/10 ht1.///ppyssoo.200.......
[14] Bjöm M. Kampa, Johannes J. Letzkus, and Greg J. Stuart. Dendritic mechanisms controlling spike- timing-dependent synaptic plasticity. Trends in Neurosciences, 30(9):46-463, Septemher 2007. ISSN 01662236. doi: 10. 10.106//..iiss 2007.0000..66.00 URL
eeseiiiiinnnabeeeeeeeeeer.... wtw.avviiiee.oorerrreeeeeeeee com/retrieve/
pi1/500662226070001733
[15] Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. In Ad- vunces in Neurul Information Processing Systems. volume 34, pages 30365-30379. Cur- ran Associates, Inc., 2021. URL https: http://ppooeeddng.enneeeee wwt.e/es.....cpppr/////22100 /hash/ cbstract.htme
[16] Manu Srinath Halvagal and Friedemann Zenke. The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks. Nature Neuroscience, 26( 1):1906- 1915, November 2023. ISSN 1097-6256, 1546- 1726. doi: 00.1038/441993-03-0044444 Publisher: Springer Science and Business
URL https:
//nuw. wwwura.aee.........mm. www./rticcees/aeeeeeeooooocoatee http:///./..///a..a////////////0
Media LLC.
[17) Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: caatieee-llvrrinne--Cvvvaaaaa Regular- ization for Self-Supervised Learning. October 2021. URL https:://openreviev. wttp://wwweeetotmn////aa xm6YD62DUUb.
[18) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, January 2019. URL http: //ap/i/////iiv///// org/abs/18/0 037448. arXiv:1807.000 [es].
[19] Willem A. M. Wybo, Matthias C. Tsai, Viet Anh Khoa Tran, Bernd Illing, Jakob Jordan, Abigail Morrison, and Walter Senn. NMDA-driven dendritic modulation enables multitask representation learning in hierarchical sensory processing pathways. Proceedings of the National Academy of Sciences, 120(32): e2300558120, August 2023. doi: 10.1073Xpnas2330558122 URL https: http://wwwwppee www.wwpp.a........ org/doi/10 1073/pnas.. 2300558120.
[20] Samantha Debes and Valentin Dragoi. Suppressing feedhack signals visual abolishes attentional modulation. Science, 379(6631)::488477, February 2023. ISSN 0036-8075, 1095-9203, doi: 10.1 126/science.ade115555 URL https: / //ww://wwwww//// http://www...n....e//eeeeeeeeese science. org/doi/10. 1tt.://................111 ww111112//eeeeee ade1855.
[21] Dina V. Popovkina and Anitha Pasupathy. Task Context Modulates Feature- Selective Responses in Area V4. The Journal of Neuroscience, 42(33):6408-641) August 2022. ISSN 0270-6474, 1529-2401. doi: 10.1 1523/JNEUOOSLLL 1386-21.2022. URL /tt.tttpp///ww..rrrrrr....r/o/// http://ro.....r./oooooo////////. 1623/ JNEUROSCI. 1 21. 2022.
[22] Sanne Rutten, Roberta Santoro, Alexis Hervais-Adelmant Elia Formisano, and Narly Golestani. Cortical encoding of speech enhances res-vnt acoustic information. Nature Human Behaviour, 3(9)9774-987 September 2019. ISSN 2397-3374. doi: 10. 15228//41562-01-006-4 URL https: h/aps:///www.. nature. Publisher: Nature Publishing Group.
[23] Patrick Mineault, Elaine Tring, Joshua Trachtenberg, and Dario L. Ringach. Enhanced Spatial Resolution During Locomotion and Heightened Attention Mouse Primary Visual Cortex. Journal of Neuroscience, 36(24)6382- 6392, June 2016. ISSN 0270-6474, 1529-2401. doi: 10. 10.1523/NNERROCI 0430- 0430-16.2011 16.2016. URL httpps://wwt www..jneurosc...o www.oo/og//ooo/eeeeeeeeoeo /ttp//eenn////24/33/// /336222. Publisher: Society for
Neuroscience Section: Articles.
12

---

[24] Laura Busse, Jessica A. Cardin, M. Eugenia Chiappe. Michael M. Halassa, Malthew J. McGin- ley, Takayuki Yamashita, and Aman B. Saleem. Sensation during Active Behaviors. The Jour- nat of Neuroscience, 37145):10826-100
November 2017. ISSN 0270-6474, 1529-2401. doi: 1523/
10. D0.152//NEUROSCLIO 18280 17.2077. URL JNEUROSCI. 1828- 2017.
[25] Serin Atiani, Mounya Elhilali, Stephen V. David, Jonathan B. Fritz, and Shihab A. Shamma. Task Difficulty and Performance Induce Diverse Adantive Patterns in Gain and Shape of Primary Auditory Cortical Receptive Fields. Neuron, 61(3):467-480, Fehruary 2009. ISSN 0896-6273, doi: 1016// neuron. 2000.12.087. URL https: hopp:://wwwt /wwp:///wwww.... cell. cww/neuron//aeoooooooo wwtpaaasstttttooeeeee 6273 (09))000 00006-3. Publisher: Elsevier.
[26] Morgane M. Roth, Johannes C. Dahmen, Dylan R. Muir, Fabia Imhof, Francisco J. Martini, and Sonja B. Hofer. Thalamic nuclei convey diverse contextual information layer of visual cortex. Nature Neuroscience, 19(2):299 307, February 2016. ISSN 1546-1726. doi: 10. 10.18//nn.499, URL https:
www.rnature.com wtt.com/aaaaaaa httmm/rriilee//eee mn. 4197. Publisher: Nature Publishing Group.
[27] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with General Conditioning Layer, December 2017. URL http://arxiv org/ abs/1709. 07871. arXiv: (tt.v17790077777 [es].
[28] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs, March 2021. URL http://arxiv.. org/ abs/ 2003.00152. arXiv:2003.00152 [es].
[29] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. TinyTL:: Reduce Activations, Not Trainable Pa- rameters for Efficient On-Device Learning, June 2021. URL http::ttp:///ttt wttp:o///g//......... #tXiv:2007.1162 [cs].
[30] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-Shot Garameter-Efficien Fine-Tunine is Better and Cheaper than In-Context Learning, August 2022. URL http::/arxiv/
org/ /a//ab/////// 220./2205.0000 05638. arXiv:2205.0555 05638.
[31] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & Shifting Your Features: New Baseline for Efficient Model Tuning. Advances in Neural Information Processing Systems, 35: 109-123, December 2022. URI https: https::/proceeedingg neerrddiiininnnnnnssseeeee hash http:///boeeeee11111eeeeeeeeee//
neurips ec/ cc/ happpppe____ sorrr-fleese ffless/ppper//22/2
Abstract- Conf erence. html.
[32] Elad Ben Zaken, Yoav Goldherg, and Shauli Ravfogel. BitFit: Simple Parameter-efficienn Fine-tuning for Transformer-be based Masked Language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguis- ties. doi: 10. 10.118653vvll (46533/112022..co 1/2022 sch-sh URL https: /at///aaaaltttt aalaaanaanntoogyyyyoooooo org/2022 acl-short. 1/.
[33] Gido M. Nature Machine Intelligence, 1185:1115--119, December 2022. ISSN 2522-5839. doi: 10.1038/ 142256-022-02 022-00568-3
Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types incremental learning.
URL https: : w/ww://wwwww/// nature com
022 00568-3
[34] Nicolas Y. Massc, Gregory D. Grant, and David J. Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44), October 2018, ISSN 0027-8424, 1091-64900 doi: 1...773/pnas..0038330 URL
https:/ //mas. oww//oi/////eooooooooooooooo foooi//ffll////// 1073//0a33p 1803839115.
[35] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-- 15184, 2020.
[36] Abhiram Iyer, Karan Grewal, Akash Velu, Lucas Oliveira Souza, Jeremy Forest, and Subutai Ahmad. Avoiding Catastrophe: Active Dendrites Enable Multi- Learning in Dynamic Environments. Frontiers in Neurorobotics, 16:846219, April 2022. ISSN 1662-5218. doi: 00.3399/inbo1222228462111 URL http:// org/ attp//aaa222..... 2201. 00042. arXiv:2201. 22000220000004222111111 [es].
[37] Nicolas Deperrois, Mihai A. Petrovici, Walter Senn, and Jakob Jordan. Learning beyond sensations: How dreams organize neuronal representations. Neuroscience & Biobehavioral Reviews, 157:105508, February 2024. ISSN 7634. doi: 10.
URL https: h/www/ sci encedirec...
13

---

[38] Fabian A Mikulasch, Lucas Rudelt, and Viola Priesemann. Local dendritic balance enables learning. of efficient representations in networks of spiking neurons. Proceedings of the National Academy Sciences, 18850))20011251118. July 2021. doi: 0..107//pnas222219951111
[39] Fabian A. Mikulasch, Lucas Rudelt, Michael Wibral, and Viola Priesemann. Where error? Hierarchical predictive coding through dendritic error computation. Trends in Neurosciences, 46(1): 45-59, January 2023, ISSN 01662236. doi: 10.101 10166666.m 2/16//..ins.222.. www.:ts..2........m URL. https:///Sskiigghub elsevier wtt.///rr..coeeeeeeeeeeeeee
[40] Kevin Kermani Nejad, Paul Anastasiades, Loreen Hertäg, and Rui Ponte Costa, Sclf-supervised predictive lcarning accounts for cortical laycr-specificity, April 2024, URL http::tttt:t////ee http://oorrii..rr//ooooooo////// 10. 1e111111222..000 04. 24. 590916.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 770-778 2016.
[42] Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Sci- ence, ((2):123-146, 1995. doi: 10. 1080009540095555033311 URL atttpe:/dol org/10. 1080/ 09540099550039318 18. tex. bibsoubso dblp computer science bibliography, https://dblp.org tex. tex.timmst
Fri, 26 May 2017 22:53:37 +0200.
[43] Sylvestre- Alvise Rebufti, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, honotulu, HI, USA. july july 21-26, 2017, pages 5533 5542, IEEE Computer Society, 2017. doi: 10.
URL https: //doi. org/10. 11.0.119///// /Cw////C..... 201...01..5555 587. tex.bibsource: dblp computer science bibliography, https//dhll.org tex. tex.timestamp: Fri, 24 Mar 2023 00:02:54 +0100.
[44] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vish- wanathan, and Roman Garnett, editors, Advances in neural information processing systems 30: An- nual conference neural information processing systems 2017, december 4-9, 2017, long beach, CA, USA. pages 6467-6476, 2017. URL hwtpt:tttppooooeee peroocedingg.nneriii. www.neuuu...cppppeeeeeeeeee cc/paper///ccpccppcccccccce.w pppappr/201//// /2017/hash/ 587527778822b22111666777799
Abstract. httml tex.bibsource; dblp computer science bib- liography, hutp::/dblp.org tex.tim tex.timestamp: Thu, 21 Jan 2021 15:15:21 +0100.
[45] Oleksiy Ostapenko, Mihai Marian Puscas. Tassilo Klein, Patrick Jähnichen, and Moin Nabi. Learning remember: A synaptic plasticity driven framework for continual learning. In IEEE conference on computer vision and pattern recognition, CVPR 2019, long beach, CA, USA, june 16-20, 2019, pages 11321-11329. Computer Vision Foundation / IEEE, 2019. doi: 10.1 1091C./.... 20120011111 URL http: //openaccess thecvf. cww/ content cowttentcVVR________o htt01///tmm/sstpmllteeeoo Learning tt____ttttobmeer..____ Cirraptic_Platiciii CaaatiiittDDrieen__...
for Conntinaannveeeeeeeec _WF___CVR________ papperrpper...m html.
bix.bubsoreee: dblp computer science bibliography. https://dblp.org tex.timestamp:
Wed, 07
Dee 2022 23:06:27 +0100.
[46] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th international conference on learning representations, ICLR 20/9, new orleans, LA, USA, may 6-9, 2019. OpenReview.net, 2019. URI stttp://oopenevieet wwt./eetruurae id=Hkf2_ssch sC5FX. bibsbibe dbip computer science bibliography. https://ublp.org tex.timestamp: Thu, 25 Jul 2019 14:25:58 +0200.
[47] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
[48]
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances neural information pro- cessing systems 33: Annual conference on neural information processing systems 2020, NeurlPS 2020, neurips...... cc/paper/ ppaper/20200 /2020/hash/
december 6-12, 2020, virtual, 2020. URL https: http://poceedddeee b704ea2c39778f 07 www.117666774480e... Abstract html. tex. bex.bibsouree: dblp computer science bib- liography, https://dblp.oo org tex. cemmtmmtttmp:: Tue, Jan 2021 15:57:21 +0100.
[49] Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, Ronald Kemker, and Christopher Kanan. SIESTA: efficient online continual learning with sleep. 2023, 2023. URL https: httpps://///eee/e// //tpp//ppeeeee.ee/////// net/forum? id-MgDVIBWRRV. bibbuueo dblp computer science bibliography, https://bblporg tex timestamp: Thu, 01 Aug 2024 15:37:25 +0200.
14

---

[50] Yibo Yang, Haobo Yuan. Xiangtai Li, Jianlong Wu, Lefei Zhang. Zhouchen Lin, Philip Torr, Dacheng Tao, and Bernard Ghanem. Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants, August 2023, URL http:://aaaai www.awwwwwww..w...........oootee ab///3/0...///. 01746. arXiv:2308.011
[51] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilci Wang, and Dahua Lin. Learning a uni- fied classifier incrementally via rebalancing. In JEEE conference computer vision and pattern recognition, CVPR 2019, long beach, CA, USA, june 16-20, 2019, pages 831-
839. Computer Vision Foundation / IEEE, 2019. doi: ///HHooLLLaanngaeee Unified_ http:/ http://openaccess.. teeeeeessssteeteeeee thecvf. www.cccccccoocccccc __tppt______________ Classifier Incrementallyl.......... watpeeettttalyyil_eeaallllee wri__Rebalanccig..... CV_p___________ 2019 -a--apprr-... .html. tex. bixbbusueee:: dblp computer science bibliography, httpss:/dblp.orgg tex.tamet Mon, 03 Mar 2025 21:01:29 +0100.
URL
[52]
Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu. Yandong Guo, and Yun Fu. Large scale incremental learning. In IEEE conference on computer vision and pattemn recognition, CVPR 2019, long beach, CA, USA, june 16-20, 2019, pages 372-382. Computer Vision Foundation / IEEE, 2019, doi: thecvf com. cttt/etttttttcceeeeeeeee _VPR__01R11_1_ (PRR_2019/htmll
10.1109/CVPR.200 200CVVP22019.0004 1009900046. URL pestp://ppenacceeeee
Wu__argee_Scar Secaal__Incremerne Satacrementa__LLannnnnnnn CVPR_201R _operr .ttml. tex. bibsource: dillp computer science bibliography, https:/dblp.org tex. tex.timestamp: Mon, 30 Aug 2021 17:01:14 +0200.
[53] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer vision ECCV 2018 15th european conference, munich, germany, september 8-14, 2018, proceedings, part III, volume 11207 of Lecture notes in computer science, pages 144-161. Springer, 2018. doi: 10. 0077/97-33003000 11919199.9 URI. https: /idoi,org/ 10. 1007 0000/978-200 3- 030-0000000000 01219-9 9_9. bibsource: dhip computer science bibliography, https:/dbbpp.og org tex. timestamp: Mon, 30 Dec 2024 20:27:58 +0100.
[54] Francisco M. Castro, Manuel J. Marin-Jimenez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End- to-end incremental learning. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu. and Yair Weiss, ed- itors, Computer vision- ECCV 2018 15th european conference, munich, germany, september 8-14, 2018, proceedings, part XII, volume 11216 of Lecture notes in computer science, pages 241-2577 Springer, 2018. doi: 10. 1007/97778--- 030-01258-80 81_, URL https:/ //dp:. org/ 10. 10000090797880 --030- 01258- 8 15. tex. bixsoubese dbip computer science bibliography, https://dblp.org tex. texttemett:m:: Tue, 01 Apr 2025 19:06:38 +0200.
[55] Arslan Chaudhry, Punect Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer vision ECCV 2018 15th european conference, munich, germany, september 8-14, 2018, proceedings, part XI, volume 11215 Lecture notes in computer science, pages 552.-72. Springer, 2018. doi: 10. 1001/999/78-
URL htt.httt////ee /ttpa:://ddoor/rr/// 10. 1007 /978-978-8 3 030 - 01252-12222222 6_33. tex. bxbbobbsuree: dbip computer science bibliography, https://dblp.org tex. timestamt:: Tue, 14 May 2019 10:00:45 +0200.
[56] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Man-Michael Frahm, editors, Computer vision ECCV 2020 16th european conference, glasgow, UK, august 23-28, 2020, proceedings, part XX, volume 12365 of Lecture notes in computer science, pages 86-102 102. Springer, 2020. doi: 10. 1007000077333 300---3000--565 58565-5166 URL https: / aadoi..o///i1 /00 1007/978-3 3- 000-00060------0-1-8811100 58565-55 5_6 tex. bibsoure::: dhip computer science bihliography, http::/dhlp.org
tex. timestamt: Sun, 02 Oct 2022 15:59:31 +0200.
[57] Enrico Fini, Stéphane Lathuilière, Enver Sangineto. Main Nabi, and Elisa Ricci. Online continual learning under extreme memory constraints. In Andrea Vedaldi, Horst Bischof, Thomas Brox. and Jan- Michael Frahm, editors, Computer vision - ECCV 2020 16th european conference, glasgow, UK, august 23-28. 2020, proceedings, part XXVIII. volume 12373 of Lecture notes in computer science. pages 720 730.735.60 Springer, 2020. doi: 10. 10010007777--0 7-787---00--0-- 3-030-58604-11 11_43. URL https:/ sttpp:///oi... /ttp://///oooo/...../. 1007/978-2 000-08800------0 58604-1. 1_43..111111111 tex. bibsouree: dblp computer science bibliography, https:/dbblp.rg
tex. tixttimett Sun, 04 Aug 2024 19:40: 12 +0200.
[58] Zhizhong Li and Derek Hoiem. Learning without forgetting. In Bastian Leibe, Jiri Matas, Nicu Sebe. and Max Welling, editors, Computer vision ECCV 2016 14th european conference, amsterdam, the netherlands, october 11-14, 2016, proceedings. part IV, volume 9908 of Lecture notes in computer science, pages 614 629. Springer, 2016. doi: 10. 10.10077/78-3-3 77778-3-319---0 16493663-- 00-43-00337.... URL https: //do:://///.///////// http://oo..o/////////// 1007/978-3-1 3- 3-19--3-1---110 319--1199------111 0_000-00300 0_37. tex. wxxbubsouree: dblp computer science bibliography, https://bblp.org
tex. timesampt: Tue, 21 Mar 2023 20:52:16 +0100.
15

---

[59] Hyuntak Cha. Jaeho Lee, and Jinwoo Shin. Co2L: Contrastive continual learning. In 2021 IEEE/CVF international conference on computer vision, ICCV 2021, montreal. QC, canada, october 10-17, 2021, pages 9496-9505. IEEE, 2021. doi: 10.1 (0.110//CCV499922.20 01199922.201.009955 2021.00938. URL pestpp::/doi.oor 1109/ICCV48922. 2021. 00938, tex. bibsorce::: dblp computer science bibliography, https://bllp.ogg tex. tim.stmp:::: Wed, 27 Sep 2023 21:14:23 +0200.
[60] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Isabelle Guyon. Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. Vishwanathan, and Roman Garnett, editors, Advances neural information processing systems 30: Annual conference on neural information processing systems 2017, december 4-9, 2017. long beach, CA, USA, pages 2990-2999, 2017. URL httttttee///proceeeee http:////oceeeenneeeeeee wew.diinnn.niiiipp...e (aae liography, https://dblp.or tex. timestmp::::: Fri, 16 Dee 2022 08:18:46+011
new. cc/paper////cccpccccccccceee.a htpppppprr/1111 /201 ///assh/ Abstract .ttll. tex. bix.bibseeeee: dblp computer science bib-
[61] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, and others. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13): 3521-3526, 2017. Publisher: National Academy of Sciences.
[62] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995, PMLR, 2017.
[63] Joan Serrå, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th international conference on machine learning, ICML 2018, stockholmssäässun, stockholm, sweden, july 10-15, 2018, volume 80 of Proceedings machine learning research, pages 4555 45544 PMLR, 2018. www.p.ees......ee... prwss/v80 serra18a, aaaraa18a...a 8attm... tex. bibsource: dblp computer science
URL WtRLt:://pprocedaiiii .ml.. bibliography, https://dblp.org tixetamestmm:: Wed, 03 Apr 2019 18: 18:30 +0200.
[64] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, 0tss/1606.44671. 2016. URL http://arriii: org/ htt///bs/10000 04671. arXiv: 1606.0467 tex. bibsouree dblp computer science bibliography, https://dblp.og tex. ee.ttmmsttmp: Mon, Aug 2018 16:46:1 11 +0200.
[65] Arthur Douillard, Alexandre Rame, Guillaume Couairon, and Matthieu Cord. DyTox: Transformers for Continual Learning with DYnamic TOken expassion. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 9275-9285, New Orleans, LA, USA, June 2022. TEFF. ISBN
doi: 10.1109/CVPR52668.... 202.000000..22200000000 URL htttp:://eeeepppl ieee. org//occueent/988019
[66] Shipeng Yan, Jiangwei Xie, and Xuming He. DER: dynamically expandable represen- tation for class incremental learning. In IEEE conference on computer vision and pat- tern recognition, CVPR 2021, virtual, june 19-25, 2021, pages 3014-3023. Computer Vi- sion Foundation / IEEE, 2021. doi: 10.11009/VP 012.44437.202..0000.3 / //openaccess. thecvf. . www/cooo/oooce (ittreent/CVPR222///t/ Representation torrttttttttttttooeooeeeeeeeeeee cor_Class___ wwwes_lanneeeeaaeeaaaeee Incremental iewrnnignnneeannnnnnnnennnnnnn _c__________________ 2021_paper__ html. tex.bibsource: dblp computer science bibliography, https://dblp.org org tex timettast:m Mon. 18 Jul 2022 16:47:40 +0200,
URL https:
[67] Bowen Zhao, Xi Xiao. Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimina- tion and fairness in class incremental learning. In 2020 JEEE/CVF conference on computer vision and pattern recognition, CVPR 2020, seattle, WA, USA, june 13-19, 2020, pages 13205- 13214. Computer Vision Foundation / IEEE, 2020. doi: N0.11000CVPRR4200022000000 URL https://openaccess thecvf. com wtwptonnnnntteoooeeeeeeeeoo Discrimination, aataaanddaaadssaeee-aaaaeea in_ ca__alas____aaaaeeaaaeeeeeaelaea Lataas__ncrrmmettaaanne NVPPR_202/html
Maintaining.
Cc_g__CV_______ 2020__02____ 202___2220__________ .tmll. bixbbbbsurree: dbip computer science bibliography, https://dblp.org tex. timeestaamp: Mon, 16 Sep 2024
10:52:02 +0200.
[68] Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning, Advances in neural information processing systems, 32, 2019.
[69] Enrico Fini, Victor G. Turrisi Da Costa, Xavier Alameda-Pinede Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Sulf-Supe Models are Continual Learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 0611-9620, New Orleans, LA, USA, June 2022, IEEE doi: 10. 00.10099CVPR32288222222000 org/do /ttp///cccmmmt///888///////
ISBN
URL https: htt/://eeexxxeeeeeeee ieee.
16

---

[70] Divyam Madaan, Jaehong Yoon, Yuanchun Li. Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In The tenth international conference on learning representations. ICLR 2022, virtual event, april 25-29, 2022. OpenReview.net 2022. URL https: tex. tix.bibso dblp computer science bibliography, https://dblp.org tex. timestamp: Sat, 20 Aug 2022 01:15:42 +0200.
/ //openreview
[71] Dapeng Hu, Shipeng Yan, Qizhenggiu Lu, Lanqing Hong, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang. and Jiashi Feng. How Well Does Sull-Supervised Pre- Training Perform with Streamin Data?, July 2022. URL http: //apiiv.////////////e/a/ org/ abs4/ 12081. arXiv:2104. 10::22004.20811 [es].
[72] Sungmin Cha, Kyunghyun Cho, and Taesup Moon. Regularizing with aecud-nneativee for continual self-supervised learning. In Forty-first international conference on machine learning, 2024. URL https:// net. met//forue http://rummmddxjxxjoo
[73] Quang Pham, Chenghao Liu, and Steven C. H. Hoi. DualNet: Continual learning. fast and slow. In Marc' Aurelio Ranzato. Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan, editors. Advances in neural information processing systems 34: Annual confer- ence on neural information processing systems 2021, NeurlPS 2021, december 6-14, 2021, vir- tual, pages 16131-16144, 2021. URL. https:/ http:////poeeeeg...eeeee
httppp.cc/ppp/////e/eeeeep 1/paper/20211 /221//ashh (Galffa8Baa55c333aa46BBBBB 2tt.33aroe - Abstract html.. tex. tex.bibsource: dblp computer science bib- liography, https://bllp.og tex. texettmtttmm:: Tue, 03 May 2022 16:20:48 +0200.
[74) Chi Ian Tang. Lorena Qendro, Dimitris Spathis, Fahim Kawsar, Cecilia Mascolo, and Akhil Mathur. Kaizen; Practical self-supervised continual learning with continual fine-tuning, In IEEF/CVF winter conference applications of computer vision, WACV 2024, waikoloa, HI, USA, january 3-8, 2024, pages 2829-2838. IEEE, 2024. doi: 10. 109/WACV57700 2024.00282. URI. htt.h:tt////att /ttp:://///oo////000//// org/10, 1109/WACV57701. 2024. 00282. bixbbibso dblp computer science bibliography, https:/dbllpoogg tex. timessapttmm: Wed, 17 Apr 2024 11:21:26 +0200.
[75] Alex Gomez-Vill Villa, Bartlomiej Twardowski, Kai Wang, and Joost Van de Weijer. Plasticity-optimiie complementary networks for unsupervised continual learning. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pages 1690-1700, 2024.
[76] Xiaofan Yu, Tajana Rosing, and Yunhui Guo. Evolve: Enhancing unsupervised continual learning with multiple experts. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2366 23776,23772222 2024.
[77] Alex Gomez-Vilke Bartlomiej Twardowski, Lu Yu, Andrew D. Bagdanov, and Joost Van De Weijer. Continually Learning Self-Supervised Representations with Projected Functional Regularization. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), pages 3866- 3876, New Orleans, LA. USA, June 2022. IEEE, ISBN 978- 87399-----79----- doi: 10.1 5044 2022.00432. URL https: //1ee////eeeeeeeeee//// leee. . /tt////ccmentt//caeeee
[78] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based complementary learning system. In The tenth international conference on learning representations, ICLR 2022, virtual event, april 25-29, 2022. OpenReview.net, 2022. URL wiiiiidrrsrrETTT.. tex. bibsouee:: dblp computer science bibliog-
https:/ /tttps://ppnrrevie wwweiiiwweneetteee raphy, https://dblp.org tex. timestap::: Sat, 20 Aug 2022 01:15:42 +0200.
[79] James L.. McClelland, Bruce L.. McNaughton, and Randall C. O'Reilly. Why there are complementary lcarning systems in the hippocampus and neocortex: insights from the successes and failures of con- nectionist models of learning and memory. Psychological Review, 102(3):419 July 1995. ISSN 0033-295X. doi: 10. 1037/0033-2952 102.3.41..
[80] Dharshan Kumaran, Demis Hassabis, and James I.. McClelland, What lecrning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated. Trends in Cognitive Sciences, 20(7): 512-534, July 2016. ISSN 1364-6613, 1879-307X. doi: 10. 10.11166...tis........ 2016. 0040..0...0000000 URL https: www. cell. Elsevier.
30043-2.
Publisher:
[81] Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat. Salman Khan, and Fahad Shahbaz Khan. Orthogonal Projection Loss, March 2021. URI. http://arxiv.org// wttp://////aaa....a/////// 14021.
arXiv:2103.144 14021..
[82] Vardan Papyan. X. Han, and David L. Donoho. Prevalence of Neural Collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 117(40):24652- 24663, October 2020. ISSN 0027-8424. 0991-6490. doi: 101073/pnaa.20155591111 URL http: / //arxiv. org/abs//2000 2008. 08186. arXiv:2008. 008888888888 186 [cs].
17

---

[83] Mathilde Caron, Ishan Misra, Julien Mairal. Priya Goyal. Piotr Bojanowski. and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Mien-TTinn Lin, editors, Advances in neural informa- tion processing systems 33: Anmual conference on neural information processing systems 2020, NeurlPS 2020, december 6- 12, 2020, virtual, 2020. URL https: https/////cceddees/ee neurips. cccccccpp/.oo...cc cccpappr/2000 /0200 00e602381711115 7411442281c1ccc Abstract html. tex. bibsoure: dblp computer sci- htmm..
ence bibliography, hupp://dblp.ogg tex. wwwestmst:::: Tue, 19 Jan 2021 15:57:31 +0100.
[84) Ting Chen, Simon Kornllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[85] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bi- lal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self-supervised learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in neural information processing systems 33: Annual conference on neural information processing systems 2020, NeurlPS 2020, de- cember 6-12, 2020, virtual, 2020. URL https: / /ptpppeeiooeec...///e neurips. cc/paper/ 22220/hashs 13ada80d5c4ee700 (at.eeee0042b177beeeeee (222992222---e Abstrac .html. tex. bibbuiboee dblp computer science bib- liography, https://dblp.co org tex. ce.ttmmst:mp:: Tue,
2021 15:57:04 +0100.
[86] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 /EFF/CVF conference computer vision and pattern recognition, CVPR 2020, seattle, WA, USA, june 13-19, 2020, pages 9726-9735. Computer Vision Foundation IEEE, 2020, doi: P0000//2222000000000
URL https://dool http::/ddooooooo orgii/or//1/ 100 1109/CVPR42600.C0 0000. 00975. tex.bibsouree dblp computer science bibliography, https:/dbblp.ogg tex. tim.stamp:: Tue, 31 Aug 2021 14:00:04 +0200.
[87] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer vision ECCV 2020 16th european conference, glasgow, UK, august 23-28, 2020, proceedings, part XI, volume 12356 of Lecture notes in computer science, pages 776-794. Springer, 2020. doi: 10. 0107799883-03-5566 11066211844400 45. https:
10. 1007 3--0//77888--0---0-0000100000003 58621-8 8_45. bibsoure:: dblp computer science bibliography, https://bblp.org tex timestamp: Fri, 27 Nov 2020 15:04:44 40100.
[88] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervidd vision transformers. In 2021 IEEE/CVF international conference on computer vision, ICCV 2021, montreal, QC, canada, october 10-17, 2021, pages 9630-9640. IEFE, 2021. doi: 10. 1 109/CCCV8992..002
.00951. URI https: / /doii org/10. /0. 1109/ICCV48922. 2021. 2021.00951, 00951. tex. bibsourcee:: dblp computer science bibliography, https://dblp.org tex. eixetimst Fri, 11
Mar 2022 10:01:59 +0100.
[89] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE conference on computer vision and pattern recognition. CVPR 2021, virtual, june 19-25, 2021, pages 19750-15754 Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR44437.220 2021. 01549. URL httpps://ppeacccess. thecef thecvf Simple_ _i siaesee__oae ddepresentation_Leariiii cotp:ing__v______111__e
Explor ing- html. bibsoure: dbip com- puter science bibliography, http:://dblp.org tex. timestamt: Mon, 18 Jul 2022 16:47:41 +0200.
[90] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th international conference on machine learning. ICML 2021, 18-24 july 2021, virtual event, vol- ume 139 of Proceedings of machine learning research, pages 12310-12320. PMLR, 2021. URL http:/ / http:://pocceednngge www.eeeeiieeerrreeeeeeeeeeeeee.e mwr. www.-preesvvvve v139/2bontarr/ html. tex. bibbouus dhip computer science bibliography, https://dblp.or org tex. timestmmt: Thu, 01 Jun 2023 15:27:03 +0200.
[91] Chun-Hsiao Ych, Cheng- Yao Hong, Yen-Chi Hsu, Tyng- Luh Liu, Yubci Chen, and Yann LeCun. Decou- pled Contrastive Learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, pages 668-684, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19809-0 doi: 10. 000779778-3-01--988999
[92] Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and Sue Yeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Information Processing Systems, 36: 36:24103-43 24128, 2023.
18

---

[93] Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna. Yonglong Tian, Phillip Isola. Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 13:18661 1861-18673, 2020.
[94] Daniel Marczak, Sebastian Cygert, Tomasz Tizcińiski, and Bartlomiej Twardowski. Revisiting supervision for continual representation learning. In European conference on computer vision, pages 181-197. Springer, July 2024. doi: 10448500aaXi..31111122
[95] Stéphane d' A'scol,,, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sa- gun. ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. Inter- national conference on machine learning, 2022:2286-22 22966 2021. ISSN 1742-5468. URL http:
cerxiv.org//aee wtwp///////////oooeeoe////////// 10697. arXiv:2103.100 10697 [cs].
[96] Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International conference on learning representations, 2022. URL https://openrevieeee net/f form? wwrrrrnnidddrdeeee K.r...............
[97] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learn- ing. In The tenth international conference on learning representations, ICLR 2022, virtual event, april 25-29, 2022. OpenReview.net, 2022. httLhttp:///ppeeree tex. bix.bbbouree:: dblp computer science bibliography, https://dblp.org tex. tex.timestamp: Sat, 20 Aug 2022 01:15:42 +0200.
htt//forumiddd881p110055 106aa..
[98] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtahar. Architecture Matters in Continual Learning, February 2022. URL http: //arxiv. wtggabs////oaoo 2////2222200000 000755. arXiv:2202.00275.
[99] Charles D. Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5):350-363, 2013. ISSN 1471003X doi: 10. 10.103//3347..
[100] Lin Zhong, Scott Baptista, Rachel Gattoni, Jon Amold, Daniel Flickinger, Carsen Stringer, and Marius Pachitariu. Unsupervised pretraining in biological neural networks. Nature, pages 1-8, 2025.
[101] Amir R. Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling Task Transfer Learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3712-3722, Salt Lake City, UT, June 2018. IEEE, ISBN 978-1-5386-6-5 ieee. org/document httpe/cummmmt//7788889/98 /878889//
9. doi: 10.1109/CVPR..0188.0 00391.
URL
[102] Steven Petersen, Benjamin A. Seitzman, Steven M. Nelson, Gagan S. Wig, and Evan Gordon. Principles of cortical areas and their implications for neuroimaging. Neuron, 112(17):283--2853, September 2024. ISSN 08966273. doi: 10. 10.10166j.neuron..2 www.ierreccm/rrtreeee//pplli http:///wweeee/00////20000003000
URL https: htttps://iinkinggtt
[103] Anthony M. Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 10(1):3770, August 2019. ISSN 2041-1723. doi: 10.1038/ s41467-019-1 117866.. URL https:///weee http:///www..u....eeee com/ http//aaccccsssssaaaee 019 -19--1
[104] Julich Supercomputing Centre. JURECA: Data Centric and Booster Modules implementing the Modular Supercomputing Architecture at Julich Supercomputing Centre. Journal of large-scale research facilities, 7(A (82), 2021. doi: 10. 1 182. URL http:/ best:://dxx /xxx doi. orggioog/..... 17811/11erff-711111 7-
[105] Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre. Journal of large-scale research facilities, fuiliiis,, 7(A 183), 2021. doi: 10. 10.17811/jisa 1/115/5ssff-7-118 188.. URL. http: / dxx doi. www.ddo....oooooooooooooooooo... org/10. 17815/ (1a11///aaff-7-1888
[106] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th international conference on learning representations, ICLR 2019, new orleans, LA, USA, may 6-9, 2019, OpenReview.net, 2019, URL https:// net/forum?ide -it.mmddddBBBleee Bilddd-kkicci"""" CQY7. tex, bibsouree dblp computer science bib- liography, hupp:///bll.org eexttmmettmm:: Thu, 25 Jul 2019 14:26:04 +0200.
[107] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim TMCL is effective in learning representations, improving over pure self-supervised approaches with sparse supervision. We underline this claim mainly by showing improvements on the main task (CIFAR-100, Table 1) and most importantly as we evaluate representations different tasks (Table 2) in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the main limitations in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Justification:
We focus on the biological correlates in this work, there are not theoretical
results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All implementation details required to re-implement our methods well as pseudo-code - are contained in the supplementary material. We use publicly available datasets and standard benchmarks from literature.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We publicly available standard datasets and submit our code. Once submitted, we will publish our code to GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Sobllic/guidee/CdeSuubbiit soteesesaabsiiioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //niss. cccccc//p. wwwpccpwpccc/eeeecc.eeceeeeeetle htt////ide///ee// "oodeSubmissiiooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All training and test details are included in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results obtained over four different pscudo-random number generator seeds and we provide the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Training infrastructure, approximate execution time and memory usage are reported in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurlPS Code of Ethics https /eteer/ineiiaee http:///eap....pppii//////////ea cc/ppublicc.hhicsssieeeee...
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: The authors acknowledge and have ensured that the conducted research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This paper describes continual learning method, highlighting potential biological correlates and therefore does not pose any striking societal impacts beyond the ones posed by fundamental machine learning and neuroscientifie research.
23

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide proper acknowledgements for the datasets, libraries and figure assets used in the supplementary materials.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs have solely been used to speed up manual implementations of the methods by per-line autocompletion. Functionally, all parts have been implemented by the authors.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp://eeeuu..eeeeeee for what should or should not be described.
26

---

A Implementation Details
Table Al: Methodological differences.
(a) Learning protocol. Number of epochs are given per session.
(b) Architectures. Architecture numbers assuming 32 x 32 image inputs and 100 classes.
Ours
[69, 721
VonViT (DyTox) [65, 95]
ResNet-18 (41)
Hrienn
Pectisining cooci
(CIFAR- 100), (wnsgeNss.N 100)
Parameters
10.7M
11.2M
opechs Onloooonaiiatonn
+ modulations
4.1M
FLOPS
1.4G
For all CIFAR-I00 experiments, we use the same class split as in CaSSLe, i.e. the same across all seeds. For the ImageNet-100 as well as the 10 session experiments (Section F), we use different class splits per seed.
For ViT experiments, we use the Adam optimizer [106] with a batch size of 256 and feedfor- ward weight decay of 0.0001 for all experiments, using per-session cosine learning rate decay with 10 warmup epochs. Con experiments are trained with a feedforward learning rate of 0.001 and a modulation learning rate of 0.01. The ResNet experiments rate of 1.0 and a modulation learning rate of 0.3 using the LARS optimizer (1) = 0.02). The Barlow Twins losses are scaled down by factor of 0.1 for Con ViT experiments, and by 0.025 for ResNet experiments. We pick the redundancy-reduution weighting factor Àøт = 0.005 for all experiments.
trained with feedforward learning
The Vin backbone has 5 'local' self-attention blocks, replacing the self-attention layers with wated-positions
eeff-attention layers, followed by 'global' self-attention block with standard self- attention. We use 12 attention heads and model dimension of 384. All images resized to input size 32 and we use a patch size of 4. The ResNet backbone conforms to the original ResNet- 18, except that the first convolution layer has kernel size of 3 and padding the first MaxPool is removed, and we remove the final MLP layers.
The code for our experiments is available at https://github com/tran--kooattma..
Modulations. The gain and bias modulations are applied to the query, key, value and output projections of all multi-head attention modules, and to both layers of the feedforward MLPs that follow the attention modules. Additionally, we modulate the positional prior of the Con ViT-speccc gated-positional self-attention (GPSA) layers, i.e. modulate the operation vhr Fij (ef. Equation 7 from d' Ascoli et [95]). Gain and bias modulations are initialized from Gaussian distribution, respectively from (L1, 0.02) and (0 0.02). We impose weight decay on the modulations with decreasing strength for deeper layers,
weight-decay == 0.4 - 0.36(cos(T 1/L)) + 1)/2
for modulations the 1-th VIT layer (out of L 6 layers). Only random horizontal flips are used as augmentations for the modulations.
Orthogonalization We virtually constrain the number of batches to the actual number of samples divided by the batch size. Let C, be the set of classes available session t. For each batch, we sample uniformly (with replacement) с C. Let X. the set of training samples of class e and X- be all other samples available at session é. Then, with probability 0.5, each class is sampled uniformly from X., or class X-c otherwise.
P(z, = 0.5 P(æ Uniform(X.) (X.)) + 0.5 P(r Uniform(X-e)).
(8)
The projector h is three-layer MLP (dimensions 2028, 2048, 2048) with ReLU activation and BatchNorm in the hidden layers. The predictor p for MI, CaSSLe and PNR is two-layer MLP (dimensions 2048, 2048) with ReLU activation and BatchNorm in the input layer. All projectors and predictors are reset at the end of each session.
Consolidation
For SupCon, we two-layer MIP (dimensions 2048, 128) with ReLU activation and NotchNorm in the input layer, and we use a temperature of 0.1 in the softmax.
As augmentations, we use
27

---

RandommesiizedCrop
size= (32),
(0ale=(008,, 1.0),
ratio=(3.0 4.0, 4.0 / 3.0),
resample-Resample BICUBIC,
ColorJitter(
brightness=0..4 contrast=0.44 ., saturation-0.22 hue=0.1, p=0.8,
Random@rayyscale(p00....
Randomtiorizontaall11pppp--- (pndonSolarrizep=0.222 2, thresholds=O.0, aaditions=0.0).
although Solarize is only applied for even views (v mod 2 = 0).
Linear probing and k-nearest neighbors For linear probing, we follow standard methodology for sule-super learning with vision transformers [107], i.c. train linear classifier top of the [CLS] tokens from the four last layers on all training samples, regardless of the labels available during continual representation learning. For ResNets, for 100 epochs using stochastic gradient descent with momentum (batch size 1024, base learning rate 0.1 with cosine decay). We do
the output of the last layer. We train
any augmentations except for random horizontal flips. For k-nearest neighbors (kNN), we obtain the representations of the last layer of the backbone instead. No augmentations used. The prediction is obtained by considering k 20 nearest neighbors, weighted by distance with temperature = 0.07.
Compute We run our experiments on the JUWELS-Booster [115] and JURECA [116] clusters at Forschungszentrum Jülich. For both systems, use single NVIDIA A10 GPU per experiment. We observe empirically that SupCon based methods have the highest GPU memory consumption (22 GB), while modulation invariance methods use 17 GB. View and state invariance require 14 GB of GPU memory. Augmentations are run on GPU and the datasets do not require image decoding, therefore CPU and RAM requirements are negligible. All runs take up 10 hours to finish the five session scenario.
B Forward and Backward Transfer
B.1
Metrics
Let task be the classification problem the classes from session i. We then define At, as the evaluation accuracy of task i at the end of training session t. We evaluate this accuracy in the task-agnostic setting, i.c. the classifier (linear or kNN) is unaware that the input data limited the task under consideration. In our FT and BT metrics, Å, the task aggosticc kNN evaluation accuracy of task on model trained from scratch using Barlow Twins on data from task i. Then, we define:
Backward Transfer
T
T
1
BT
(Ar. - A
(9)
T
T
t=i41
Forward Transfer
T
1-1
(Ars - A.)
(10)
T
1=2
28

---

Task
Task
Task 3
Task
Task
60%
SupCon
50%
SI (CaSSLe)
SupCon
40%
BT
81
FT
MI
Session
Figure A1: Forward and backward transfer of different methods. Accuracies on class- class-incremental CIFAR-100 (5 sessions) given either 1% of labels completely unsupervised (averaged over four seeds).
B.2 Forward and Backward Transfer of Different Methods
Previously, investigated the stability-plastiiity trade-off modify the strength of the modula- tion invariance term (Figure 3). We further demonstrate in the sparsely labeled learning scenario (1% labels), that modulation invariance also provides improved forward and backward transfer compared to SupCon, while SupCon surprisingly shows lower plasticity than VI (Figure Al). The introduction of state invariance shows further improvements, the combination SI and on top of VI yields the highest forward and backward transfer on most tasks except for the first task.
C Python-style Pseudocode for TMCL
# f (xs, t): forward pass of backbone with inputs xs and modulations t C_1, .... C_t: list of classes of session 1, .... def orthogonalizations (xs, ys): # sparse labeled samples at session t for step in range (orthog_steps).
c random. Sandom.sample(C (C_t, k=1)
positives random, sample c], k=batch_siza // negatives = random sample lys != cl, shaacc__siee // 2)
pos_f, neg_f = (positives, t=c), f (negatives, t=c) loss opl_loss (pos_f, neg_f)
loss. backward
update (f wwm.muatiionsssc [c])
def consolidation is.pretrain=False)) unlabeled samples at session for step in Langeccon_ssteps)
batch random. sample (xs, (obatchh_size)
views = [aug (batch) for in range [um_views)]
# view invariance
# h_vi: view-inv. projector
vi_projs [h_vi (f t=None)) for v in views]
vi_l instrasstvee__ssstvviri
# Multi-view Barlow Twins
if is. is_pretrain:
vi_loss. backward update (ftffeeffrraarrdeeeeieooe continue
if enabled: model state invariance
# f _past: frozen backbone from prev. session
h_vi_past: frozen view-inv. projector from prev. session
# p_si: state-inv. predictor with torch. ()
si si_projs_past [h_ vi vivvvppatte (f (vst(vv, t=None))) ]
29

---

si_preds_curr (viipri)r))s) si_loss mean
distill loss past) CaSSLe/PNR loss function for curr, past papeese_pred_ccuu i.projs_past)
distill_lossccuree
# modulation invariance
# h_mi: mod-inv. projector
# p_mi: mod-inv. predictor
mi tasss [[None] * batch_size] unmodulated first view
mi mi_tasks += Sandom.sammle(CC
CLt, x-batchhsize) for in views [1:]] mi _i_ps = [h_mi(f(t) t=t)) for v, t in zip(views, mi mi_tasks)]
pred
(mi p_mi(mi_rrojs_[]]
mi loss contrastive_loss (mi pred, *mi_projs[1:]) # Multi-view Barlow Twins
loss = vi_loss + si_loss + mi_loss
loss backward()
update (f feeffoorward
def train (sessions):
for session_ids (xs_unlabeled, xs_labeled, ys_labeled) sessions: if session_ide _ix == 0:
consolidation (xstttiooos_sssslll unlabelel, is (s.pprerrain-rue)
orthogonalization (xs (xssiiaaiiiooxs_aaeeeellel ys_labeled)
consolidation (xs_ttiiiiooxs___ulleeeeeeeee
D Implicit Orthogonalization via Modulation Invariance
This section seeks walk through the intuition behind the implicit orthogonalization via modulation invariance. To do so, we assume a non-incremental fashion with four classes A, B, C, D. For conceptual clarity, focus the explanation here the class centers of these respective classes.
In the orthogonalization phase of TMCL, we train modulations mA to achieve A Alm,11 [( m Cl Dl ml J. mn to achieve B B|mg 1 ( ( Cl Dimgh, J. mc achieve Clma 1 (A) [AIme: B Blma: D me): and mp to achieve D mmpp {A [mp: Blmp, C m. Here, A Alma denotes the representational vector of the class center of class A under modulation m, and similar for all others.
In the consolidation phase, our goal is to arrive at a representation space where Blp A A Llg-DDlaA o /C LDl (where Ale. : Blg. C and Dlu denote the class centers of A, B, C, D in the unmodulated network). can be seen that this will be the case, if we simultaneously achieve the orthogonality relations of point and AIp A| m Almm AAmm Almp (where A Alma.g.c.P. denotes the class center of class A respectively under modulations mA, m., mc. and mp) and similar for classes B, C. D. The contrastive objective maximizes similarity between given data sample without modulation and under modulations mA, m, mc. and m.D and therefore implicitly drives the network to a state for which Ale Alma = A Alma = Alme = A mn and similar for B, D.
We note that achieving the full orthogonality relation Alp 1B AApp-//Cooaaa AL
Llp A Clg would likely require iterating steps and 2. However, we did not seek such an iterating implementation, as focused on the continual learning setting where we identified modulation learning (step 1) with single phase of fast learning that occurs whenever new class is observed, and step with a slower consolidation phase. Under these conditions, the full orthogonality relation cannot be expected to be achieved rigorously, but as is demonstrated by our CDNV results, our TMCL. algorithm still leads representation space where the clustering of the individual classes improved.
30

---

E
Class-Distance Normalized Variance
The class-distance normalized variance (CDNV) is defined as
intre-class collapse
Var(Z Var(Z (c)
CDNV
(11)
[ ( (Z(())
c, c EC oft!
inter-class distance
where Z(o)
1z (c)
0
E NND is batch of backbone representations (i.e. z.
f(x; W. 0)) all class-c samples in the CIFAR-I00 test split, and the set of CIFAR-100 classes (lower is better).
F Split-CIFAR-100 with 10 sessions
demonstrate that TMCL also effective the scenario of sparsely supervised continual learning with 10 sessions, i.e. 10 classes per session (Table A2).
Table A2: Semi-supervised continual representation learning. Final all-vs-all accuracy class- incremental CIFAR-100 (10 sessions) given either 1% of labels or completely unsupervised, averaged over four seeds (#t denotes the standard deviation).
Method
kNN
linear
SupCon + SI (PNR)
34.7=006
20.9+1.4
CE
39.2
28.2
VI
55.5
50.3
MI (TMCL)
57.7=002
52.2
SupCon
55.1 ::0.3
49.9
SI (PNR)
57.4.00.5
54.0
SupCon
57.5
54.0
+ CE
57.4
54.2
MI
58.7+t03
54.6
G
Datasets
Our analysis focuses on the CIFAR-100 dataset [117] and the ImageNet-1000 100 dataset [111]. For the
transfer learning experiments, we perform kNN evaluation on Aircraft [118], CIFAR-10 [117], CUBirds [122], DTD [109), EuroSAT [113], GTSRB [114], STL-10 [110], SVHN [119], and
VGGFlower [120].
H Further acknowledgements
We implement our methods based on PyTorch [108] with the Lightning framework [112]. For the
augmentations on CIFAR-100, we used kornia [121]. Backbone implementations are adapted from the timm library [123]. Data loading and augmentations for the ImageNet- 100 experiments are implemented via NVIDIA DALI (https://gil /ttt::b//githbb....
thub. com/DVIDIA/DALII, For the figures, we relied on icons designed by OpenMoji (License: CC BY-SA 4.0).
31

---

Supplementary References
[108] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesen- sky. Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary VeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcamo, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang. Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Trans- formation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS '24). ACM, April 2024.
[109] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[110] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[111] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[112] William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019.
[113] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266,
2019.
[114] Sebastian Houben, Johannes Stalkkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-worke images: The German Traffic Sign Detection Benchmark. In International Joint Conference Neural Networks, number 1 1288, 2013.
[115] Stefan Kesselheim, Andreas Herten, Kai Krajsek, Jan Ebert, Jenia Jitsev, Mehdi Cherti, Michael Langguth, Bing Gong, Scarlet Stadtler, Amirpasha Mozaffari, et al. Juwels booster-a supercomputer for large-scale ai research. In International Conference High Performance Computing, pages 453-468. Springer, 2021.
[116] Dorian Krause and Philipp Thömngg Jureca: modular supercomputer at jülich supercomputing centre. Journal of large-scale research facilities JLSRF,
2018.
[117] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[118] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013.
[119] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[120] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[121] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Kornia: an open source differen- tiable computer vision library for pytorch. In Winter Conference on Applications of Computer Vision, 2020.
32

---

[122] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 I dataset. 2011.
[123] Ross Wightman. PyTorch Image Models.
33


Reviews:
[{"review_id": "9FL4G33cWE", "reviewer": "Reviewer_fD14", "summary": "This paper proposes a biologically inspired approach for semi-supervised continual learning, called Task-Modulated Contrastive Learning (TMCL). Inspired by the segregated cortical architecture of top-down and feedforward pathways, the method uses modulations (learned affine transformations) to integrate sparse supervisory signals into a contrastive learning framework. The key insight is that top-down modulations learned from few labeled examples can be consolidated into a modulation-invariant representation space through predictive coding-like contrastive losses. This allows learning new concepts with minimal forgetting, even in class-incremental learning without task identity. The method is extensively evaluated on CIFAR-100 under varying supervision regimes and transfer learning tasks.\n\nThe paper is generally well-written.", "questions": "1- can authors add a discussion on how to scale TMCL to larger datasets or class hierarchies?\n\n2- can authors provide a visualization of representation space dynamics across sessions (e.g., UMAP plots) to support the modulation-invariance hypothesis?\n\n3- can authors discuss robustness to label noise or incorrect modulations?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:The orthogonal projection loss (OPL) to train class-specific modulations, and the consolidation of those modulations using modulation-invariant contrastive learning (MI), is a contribution.TMCL performs competitively or better than existing state-of-the-art methods in low-label or no-label continual learning settings.\n\nWeaknesses: -Since a unique modulation is stored per class, scaling to tasks like ImageNet-1K may pose challenges in both memory and management. This is briefly mentioned as a limitation, but a forward-looking strategy (e.g., shared modulation generators) would be helpful.\n\n-All experiments are performed on CIFAR-100. This dataset is limited in visual diversity and resolution. Results on higher-resolution benchmarks like miniImageNet or ImageNet-Subset would improve generalizability.", "was_revised": true, "final_justification": "The authors have thoroughly addressed my concerns and I appreciate their detailed responses to other reviewers' concerns. \n\n- The authors' response to the comparison of TMCL with other benchmark models clarifies the contribution of the work. Like reviewer vrza, I believe that having this biologically inspired model doesn't make other biologically inspired models less interesting. Moreover, adding this biological details expands the impact of this work to other fields such as systems neuroscience.\n\n- Biological brains effortlessly solve continual learning problem. TMCL has comparative performance to other methods using brain inspired algorithm. This is done through orthogonalization via invariance learning inspired by cortical predictive learning. \n\n- Biological brains face uncertainty even when they are provided with labeled feedback. The authors have added new analysis assessing the robustness of TMCL to label noise which shows comparative advantages over previous methods such as SupCon.", "rebuttal": "We appreciate that the reviewer highlights the novelty of orthogonalizing modulations and the consolidation thereof via modulation invariance. As suggested by most reviewers, we now provide results on ImageNet-100, showing stronger improvements of TMCL. We will further strengthen our manuscript thanks to the reviewer's discussion on scaling classes and robustness to label noise.\n\n### 1. New ImageNet-100 experiments show strong improvements over baselines. (W2)\nWe have now extended our empirical study to ImageNet-100 (5 sessions). Our experiments (cf. Tables 1, 2) show that our method (+ MI) outperforms all other methods on 10% and 1% labels, with a significant improvement of +3.8pp and +2.3pp given 10% and 1% of ImageNet-100 labels respectively on linear evaluation.\n\n### 2. Per-class modulations are cheaper than sample replay, and replacing them with a modulation generator is a promising avenue for future work. (W1, Q1)\nIn principle, the number of classes that can be accommodated in TMCL is unlimited. Of course, in our current setup, there is a practical limit in terms of memory. Analyzing the memory consumption (Table A1b) shows that storing modulations for 100 classes requires storing approximately 4.1M parameters, compared to 10.7M for the storing the base ConViT model (without modulations). This means that modulations for 250 classes take up as much memory as the base model, marking the break-even point with SI methods (CaSSLe, PNR), which store a copy of this base model.  Furthermore, this equates to around 16MB per 100 classes, while e.g. storing 500 ImageNet images (on average) already takes 50MB.\n\nHowever, an avenue of particular interest is the generation of modulations with a modulation generating network. In such a setup, memory size is fixed, and the number of classes that can be accommodated is determined by the expressivity of the modulation generator. In future work, we aim to explore such a setup, as this is anyway closer to how we hypothesize that modulations of sensory networks are being generated in the brain.\n\nAn important consequence of having such a modulation generator, is that TMCL can be extended to incorporate class hierarchies. Modulations for narrow classes (for instance, types of birds) could be consolidated into a modulation for a more general class (\"bird\") through TMCL, which in turn could be consolidated into a yet more general class (\"animal\"). Recognition of sensory percepts could then proceed along this hierarchy (i.e. \"type of object? - animal\" --> \"type of animal? - bird\", \"type of bird? - sparrow\", where each of these questions represents a different modulation applied to the sensory system). Such a process, we believe, is close to how biological perception of unfamiliar objects works.\n\n### 3. TMCL is very robust to label noise, while other supervised methods deteriorate. (Q3)\nThe suggestion of the reviewer piqued our interest, and we ran an experiment testing the robustness of TMCL w.r.t. label noise. The state-of-the-art supervised continual representation learning method (`CE`) proposed by Marczak et al. [1] underperforms unsupervised Barlow Twins (`VI`) as label noise increases past 40%. Similarly, they show that at least 30% of the supervised data is necessary in order to compete with `VI`, while TMCL (via `MI`) does not deteriorate in regimes with significantly fewer labels.\nIn these new experiments, we replace k% of labels with random ones (Table 3). We observe that `MI` is robust to almost completely random labels, and is the only loss that does not deteriorate performance below the unsupervised `VI` baseline. We note that this is consistent with our ablations where we use random or untrained modulations, since for high label noise levels, the modulations would carry no semantic meaning and just represent an additional noise source to which the self-supervised objective learns to become invariant. We note furthermore that the supervised objective never *directly* influences the unmodulated representation space, opposed to e.g. SupCon, where with high label noise the supervised learning has a direct detrimental effect on the representations. We will add these results with label noise to the appendix of the camera-ready version.\n\n### 4. Class-Distance Normalized Variance (CDNV) quantifies representation space dynamics, suggesting orthogonalization via modulation invariance. (Q2)\nThe reviewer requests a visualization of the representation space dynamics to support the hypothesis that modulation invariance consolidates the class orthogonalizations introduced by the modulations. Unfortunately, we are unable to attach these visualizations in this year's rebuttal modality. However, our analysis of the evolution of the CDNV demonstrates that class clustering improves during the consolidation phase with `MI`, but not with pure `VI` or any other methods.\n\n----\n**Table 1:** Class-incremental ImageNet-100, 5 sessions (linear eval., averaged over 3 seeds)\n| **Method**                               | **100% labels** | **10% labels** | **1% labels** | **unsup.** |\n| ---------------------------------------- | --------------- | -------------- | ------------- | ---------- |\n| SupCon                                   | 64.0            | 48.7           | 33.6          | -          |\n| &nbsp;&nbsp; + SI (PNR)                  | 64.1            | 48.5           | 33.5          | -          |\n| CE                                       | 66.0            | 50.1           | 35.2          | -          |\n| VI                                       | -               | -              | -             | **59.7**   |\n| &nbsp;&nbsp; + SupCon                    | 66.6            | 61.4           | 60.2          | -          |\n| &nbsp;&nbsp; + CE                        | 64.5            | 61.4           | 60.3          | -          |\n| &nbsp;&nbsp; + **MI** (ours, TMCL)       | 64.5            | **63.5**       | **62.0**      | -          |\n| &nbsp;&nbsp;+ SI (PNR)                   | -               | -              | -             | 59.6       |\n| &nbsp;&nbsp;&nbsp;&nbsp; + SupCon        | **67.0**        | 60.2           | 43.9          | -          |\n| &nbsp;&nbsp;&nbsp;&nbsp; + CE            | 64.5            | 60.3           | 59.4          | -          |\n| &nbsp;&nbsp;&nbsp;&nbsp; + **MI** (ours) | 63.8            | 62.7           | 61.7            | -          |\n\n**Table 2:** Class-incremental ImageNet-100, 5 sessions (kNN eval., averaged over 3 seeds)\n| **Method**                               | **100% labels** | **10% labels** | **1% labels** | **unsup.** |\n| ---------------------------------------- | --------------- | -------------- | ------------- | ---------- |\n| SupCon                                   | 57.5            | 40.2           | 25.6          | -          |\n| &nbsp;&nbsp; + SI (PNR)                  | 57.6            | 40.7           | 25.7          | -          |\n| CE                                       | **62.8**        | 44.0           | 26.1          | -          |\n| VI                                       | -               | -              | -             | 50.5       |\n| &nbsp;&nbsp; + SupCon                    | 57.7            | 52.4           | 50.9          | -          |\n| &nbsp;&nbsp; + CE                        | 56.2            | 52.0           | 51.0          | -          |\n| &nbsp;&nbsp; + **MI** (ours, TMCL)       | 56.0            | 55.1           | 52.1          | -          |\n| &nbsp;&nbsp; + SI (PNR)                  | -               | -              | -             | **51.7**   |\n| &nbsp;&nbsp;&nbsp;&nbsp; + SupCon        | 59.3            | 52.4           | 51.9          | -          |\n| &nbsp;&nbsp;&nbsp;&nbsp; + CE            | 56.4            | 52.5           | 51.6          | -          |\n| &nbsp;&nbsp;&nbsp;&nbsp; + **MI** (ours) | 56.3            | **55.3**       | **52.7**      | -          |\n\n**Table 3:** Label noise (linear evaluation, averaged over 4 seeds)\n\n| Method             | 30% noise | 50% noise | 90% noise | 99% noise |\n| ------------------ | --------- | --------- | --------- | --------- |\n| CE                 | 57.2      | 54.4      | 8.4       | 7.9       |\n| SupCon             | 58.2      | 56.8      | 48.7      | 47.0      |\n| *VI (unsup.)*      | *59.3*    | *59.3*    | *59.3*    | *59.3*    |\n| VI + SupCon        | **60.9**  | 60.0      | 58.3      | 58.3      |\n| VI + CE            | 59.3      | 59.0      | 58.3      | 58.3      |\n| **VI + MI (ours, TMCL)** | 60.6      | **60.3**  | **60.3**  | **59.4**  |\n\n\n### References\n[1] Marczak et al., Revisiting supervision for continual representation learning. ECCV 2024."}, {"review_id": "hLV5S5dQW0", "reviewer": "Reviewer_4K39", "summary": "The paper presents a biologically inspired ML method for continual learning. The paper combines contrastive learning and orthogonal representation learning to mitigate forgetting and learn from partially labelled data in a largely unsupervised data stream. Results on the CIFAR-100 dataset show improvement over other baseline CL methods.", "questions": "Please see the weaknesses section for my questions.", "limitations": "Limitations have been discussed adequately. However, positive and negative impacts have not been discussed. This should be added in the paper as there can be a variety of impacts that can be discussed for the proposed method.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The paper covers the biological concepts quite well and provides a strong motivation for biologically inspired learning methods for continual learning. \n2) The paper considers the problem of semi-supervised continual learning, where most of the data is unlabelled, which is closer to realistic settings.\n3) A large number of experimental results are reported on CIFAR-100 that show improvement compared to a few other models.\n\nWeaknesses:\n1) While the model does combine some well-known concepts into a new method, how does it compare to other biologically inspired methods? For example, SI [b], FearNet [a], CBCL [c], [d, e], to name a few. There have been a lot of CL methods proposed that have been biologically inspired. How does the current method compare to those methods? And more importantly, what is the novelty of the proposed approach compared to prior works?\n2) The proposed work has only been compared against a few baselines from the literature. How does this model compare against well-known CIL methods, such as replay-based, regularization-based, generative replay-based, etc., methods?\n3) From tables 2 and 3, I do not see significant changes in performance by adding different modules to VI. This might mean that many of these modules are not contributing much to the overall performance, and it could be because of using the ViT architecture compared to a CNN, such as ResNet-18. How does the model perform with ResNet-18? The paper states in L202-23 that ViT was chosen because of its ability to mitigate forgetting. However, ViT is not the main contribution of this work, so it would be interesting to see the contribution of the backbone compared to the proposed biologically inspired modules.\n\n[a] FearNet: Brain-Inspired Model for Incremental Learning, ICLR 2020.\n[b] Zenke, F., Poole, B., & Ganguli, S. (2017, July). Continual learning through synaptic intelligence. In International conference on machine learning (pp. 3987-3995). PMLR.\n[c] Ayub, A., & Wagner, A. R. (2023). Cbcl-pr: A cognitively inspired model for class-incremental learning in robotics. IEEE Transactions on Cognitive and Developmental Systems, 15(4), 2004-2013.\n[d] Madireddy, S., Yanguas-Gil, A., & Balaprakash, P. (2023, November). Improving performance in continual learning tasks using bio-inspired architectures. In Conference on Lifelong Learning Agents (pp. 992-1008). PMLR.\n[e] Wang, L., Zhang, X., Li, Q., Zhang, M., Su, H., Zhu, J., & Zhong, Y. (2023). Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence, 5(12), 1356-1368.", "was_revised": true, "final_justification": "I have engaged in extensive discussions with the authors and looked at other reviewers' discussions. I noticed that all the reviewers are in agreement to accept the paper, and agree that the biologically inspired concepts presented by the paper are novel and interesting for continual learning. While I agree that the paper is interesting and these concepts are novel, it is still unclear why we need another paper that provides an interesting idea but fails to clarify how it might advance the continual learning field as a whole and provide a viable direction to deal with the fundamental problems in continual learning. \n\nDue to these reasons, I am inclined to reject the paper, though I would not strongly oppose it being published.", "rebuttal": "We thank the reviewer for their extensive feedback. We believe the reviewer's concerns regarding novelty and comparison to previous methods stem from misunderstandings. We will improve our manuscript to alleviate the confusion, and we invite the reviewer to read the updated introduction to Section 3 (`XmrC`, Sec. 2). We fully address performance concerns with new experiments on ImageNet-100 and previous results on ResNet-18.\n\n### 1. TMCL is not just a combination of prior \"modules\". (W1)\nAs to our knowledge, the concept of learning to be invariant to modulations - carrying high-level \"top-down\" information - is novel in both neuroscience and machine learning. We acknowledge that at first glance, Table 1 in the manuscript suggests TMCL to be a combination of prior methods. However, combinations with `SI` solely suggest that `MI` is not orthogonal to prior approaches. **The main contribution is the introduction of consolidation of previously learned modulations via modulation invariance (`MI`), in combination with view invariance (`VI`), i.e. `TMCL = VI + MI`**. We will alleviate this confusion by simplifying Table 1 in the manuscript and moving complex combinations to a separate table.\n\n### 2. TMCL is primarily a machine learning analogue to our hypothesized cortical learning mechanism, but also provides significant improvements in learning *representations* continually. (W1, W2, W3)\nFrom previous work on cortical learning, in particular the role of NMDA-driven top-down modulations impinging on distal dendritic branches and therefore shaping feedforward computations, we derive **a biologically inspired machine learning analogue**. \nTMCL aims to tackle continual *representation* learning in the spirit of CaSSLe [1] and PNR [2]. While these works - and our work - evaluate on class-incremental benchmarks, **these are not *class-incremental learning* algorithms**, since the readout (either linear or kNN) is obtained on all available data and is hence considered part of the evaluation. In Section 6, we discuss the omission of previous class-incremental approaches, which have already been demonstrated to be inferior to CaSSLe [1].\n\n### 3. Performance improvements are significant with weaker ResNet baseline, and on the more complex ImageNet-100 dataset. (W3)\nWe would like to stress that **we do not primarily aim to outperform state-of-the-art methods**, but to show that our biologically grounded method is at least competitive to the state-of-the-art (as Reviewer `vrza` identified).  However, as the reviewer correctly points out, the changes in performance are less significant since we opted to focus on the stronger ConViT baseline.\n\n**Still, we point out that we do in fact provide results on ResNet-18 in the manuscript.** \nWe observe a significant improvement of TMCL (`VI + MI`) over `VI` on ResNet18 (**+4.7pp in linear evaluation**, cf. Table 5 in the submitted manuscript). However, the improvements of `VI + SI (PNR)` are stronger with ResNet18 (+6.5pp), which we attributed to the susceptibility of ResNets to forgetting, as the reviewer points out. We want to highlight that these results suggest that TMCL works on both ViT and ResNet architectures.\n**Significant improvements over `VI` are also observed on ConViT on ImageNet-100 (Rebuttal to `fD14`, Section 1, Tables 1 & 2),** as TMCL outperforms all other methods in the semi-supervised settings, improving over the `VI` baseline by +3.8pp (10% labels) and +2.3pp (1% labels).\n\n### 4. Suggested methods [a,c-e] are not comparable because they aren't primarily representation learners. We instead provide results to a comparable biologically inspired method. (W1)\nWe furthermore thank the reviewer for pointing out the references to these continual learning approaches that draw inspiration from biology **(weakness 1)**. It was not fully clear to us whether the reviewer is most interested in comparing the performances achieved by these various models, or rather in discussing their biological merits. As for performance, we considered SI [b] to be similar to elastic weight consolidation (EWC), which was evaluated in the CaSSLe paper [1] and underperforms both theirs and our `VI + MI` results (Table 4). FearNet [a], CBCL [c], and NNA [d] use a feature extractor pretrained on ImageNet. These approaches can thus not be considered representation learning algorithms, as they focus on learning a readout (or readout network) on top of a powerful pretrained model. Finally, [e] appears to only report task-incremental performance numbers. Instead, we compare to CLS-ER as a strong, biologically-inspired continual learning method (Table 4). TMCL outperforms CLS-ER given all labels, while CLS-ER significantly deteriorates in scarce supervision regime (10% labels).\n\n### 5. Suggested biologically-inspired methods [a-e] are complementary to TMCL. (W1)\nAs for biological relevance, we note that the biological mechanics these papers aim to leverage in a continual learning setup are complementary to ours. SI [b] focuses on preserving synaptic weights important for prior tasks, while FearNet [a] proposes hippocampal and prefrontal modules for generative replay. Finally, CBCL [c] uses the attractive cognitive idea of storing pseudo-class centroids based on a clustering algorithm. Our paper describes a novel mechanism based on the spatial organisation of the cortical microcircuit. The remarkable performance of biological systems in processing continual data streams is likely a combination of several of such mechanisms. Therefore, a systematic study investigating the merits of combining several of these biologically plausible continual learning mechanisms would certainly be interesting, but is outside the scope of the present work. It should be noted, however, that when one aims to extend generative replay mechanisms, or centroid based mechanism, to full sensory representation learning (i.e. beyond the learning of a shallow readout), one naturally arrives at a view where these mechanisms modulate the neurons in the sensory network. Therefore, using these approaches in some way as the input to a modulation generation network is a promising extension of our current work.\n\n### 6. CaSSLe / PNR baselines cited in the manuscript are state-of-the-art for continual representation learning. (W2)\nFini et al. performed studies on continual representation learning on a set of standard class-incremental approaches, namely \"prior-focused regularization (EWC), data-focused regularization (POD, Less-Forget), and rehearsal-based replay (ER, DER) methods\", and found these to be inferior to CaSSLe. For more details, refer to [1]. Therefore, similarly to [2], we assume CaSSLe [1] / PNR [2] to be state-of-the-art, but will provide results from [1] via Table 4 in the final version of our paper. Again, we want to highlight that these methods are complementary to TMCL.\n\n### 7. Impact Statement (regarding \"Limitations\")\nWith TMCL, we develop a novel, biologically-inspired algorithm for continual representation learning. Our work elucidates potential learning mechanisms in the cortex, contributing to the neuroscientific understanding of predictive coding as a canonical learning rule and the biological machinery behind the orthogonalization of stimuli. At the same time, TMCL allows for semi-supervised representation learning, e.g. in robotics and edge devices in a more realistic continual learning setting in which unlabelled samples are vast and labels are sparse. We do not find any direct negative impacts beyond the ones posed by fundamental machine learning and neuroscience research.\n\n---\n**Table 4:** Comparison to CLS-ER and prior class-incremental approaches on ResNet-18 and CIFAR-100, averaged over 4 seeds. We adapt further results from [1].\n\n| Method (R18)                                   | linear   |\n| ---------------------------------------------- | -------- |\n| CLS-ER                                         | 56.7     |\n| CLS-ER (10%)                                   | 48.8     |\n| VI                                             | 53.4     |\n| &nbsp;&nbsp; + **MI** (ours, TMCL)             | 58.1     |\n| &nbsp;&nbsp; + **MI** (ours, 10% labels, TMCL) | 56.7     |\n| &nbsp;&nbsp; + ER [1]                          | 54.6     |\n| &nbsp;&nbsp; + DER [1]                         | 55.3     |\n| &nbsp;&nbsp; + LUMP [1]                        | 57.8     |\n| &nbsp;&nbsp; + Less-Forget [1]                 | 56.4     |\n| &nbsp;&nbsp; + POD [1]                         | 55.9     |\n| &nbsp;&nbsp; + SI (CaSSLe) [1]               | **60.4** |\n\n#### References\n[1] Fini et al., Self-supervised models are continual learners. CVPR 2022.\n[2] Cha et al., Regularizing with pseudo-negatives for continual self-supervised learning., ICML 2024."}, {"review_id": "0m9gYJeDIR", "reviewer": "Reviewer_XmrC", "summary": "Motivated by the continual learning ability of biological brains, this submissions considers the problem of learning good representations from mostly unlabelled data. The authors propose a representation learning method that uses a contrastive loss to learn modulations that orthogonalize representations. These modulations are then used to consolidate the data into the weights of the neural network. To evaluate the proposed method, the authors compare their method on different scenarios using CIFAR100, including on some downstream tasks.\n\n\nDecision: My current assessment is a weak rejection with room for improvement. While the submission provides some insight and develops a generally applicable method for improving representation quality, it is held back by some clarity issues and experiments which are not entirely convincing.", "questions": "- The technical meaning of the term \"modulation\" and \"top-down\" is never defined. In equation (1) and (2) it seems to be defined as additional elementwise multiplicative and additive parameters, which play a specific role in representing task information. But I would prefer to see the difference from the usual weights more explicit. \"BatchNorm [...] is equivalent to learning affine modulations\" This adds to my confusion because the weights for batchnorm are treated as elementwise trainable weights, carrying no semantic meaning about task information.\n- I see some discussion in reference to parameter-efficient fine tuning methods like refs 30 and 31. But what is the relationship between this method of adapting parameters to task and approaches that factorize matrices, such as low rank adapters?\n- \"modulations (‘How did we solve the task?’), and do not investigate methods with exemplar replay.\" -> it would still be interesting to compare results to replay methods for completeness.\n- Are the modulations and weights trained in completely independent phases?\n- If there are two phases, how is the data split between the phases? How can you train with the contrastive loss in the \"task-learning\" phase if it seems to require orthogonalization between classes, when the data is mostly unlabelled?\n- I like the idea of orthogonalizing the modulations of each class, but I wonder if this representation may impede effective generalization in cases where it is not needed, such as in a single task-setting? (It is fine if this is the case, but it is not clear to me one way or another)\n- Line 172: what is the rationale for using the average of the views here?\n- Table 1: it is comparatively less clear what the set of modulations are, how they are sampled, and why there are an equal number of modulations (+1 for unmodulated) and augmentations (since both are indexed by $v$. Does every single instantiation of an augmentation (e.g., rotate by x degrees), correspond to a unique modulation m_x?\n- Some theoretical justification for this approach would help clarify what the contribution of this method is beyond the presented motivation.\n\n### Minor Comments\n- Why is the term \"session\" used periodically when elsewhere the term \"task\" is used? They seem interchangeable, and should be standardized for clarity.\n- line 177 (typo): appling -> applying\n- Table 2 and 3 would be better served by emphasizing the proposed method, such as adding (Ours), to make the results stand out.\n- $\\mathcal{L}_{CL}$ is mentioned in the text but never defined, is it equivalent to $\\mathcal{L}_{BT}$?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The idea of learning transformations to orthogonalize the representation is compelling and seems effective in the experiments presented.\n- Motivation: the authors make it clear that their motivated by understanding how learning algorithms can achieve the same continual learning ability as biological brains.\n### Weaknesses\n- While the experiments cover many different scenarios, the base dataset in all cases is CIFAR 100 can be limiting in its complexity and difficulty compared to ImageNet variants like tiny-ImageNet.\n- It is not detailed exactly what this algorithm does. The loss and architecture are clear, but it is never put together in a way that makes it clear how the modulations are being learned and used.\n- Minor but relevant given the motivation: Some choices in the submission add up to undercut the original biological motivation. For example, rather than a streaming setting as originally mentioned, the experiments are in the batch setting. Rather than alternating consolidation and task-learning, the model is given special phases in each task, one at the beginning for consolidation and then one for consolidation.", "was_revised": true, "final_justification": "After the rebuttal, the authors have included additional results on a second dataset, which addresses one of my primary concerns. In their individual reply, the authors also clarified several points regarding the novelty of their approach (in consolidating different orthogonalizations). They also provided additional justification for the choices that they made during experiments (which are a departure from their original motivation) as well as described potential mechanisms to explain the effectiveness of learning modulations for orthogonalized representations.\n\nGiven the thorough reply, I will increase my score to a 5. However, the authors should update the submission with these additional details in a camera ready.", "rebuttal": "We appreciate that the reviewer finds our biologically grounded idea of continually consolidating modulations into an orthogonalized representation space compelling, and thank the reviewer for their inquisitive remarks. We have reworked the introduction to Sec. 3 to improve clarity of the method early on in the paper, and address experimental concerns by providing new ImageNet-100 results.\n\n## 1. New ImageNet-100 experiments show larger improvements. (W1)\nOn ImageNet-100, TMCL outperforms all other methods in the semi-supervised setting by +3.8pp (10% labels) and +2.3pp (1% labels), cf. rebuttal to `fD14` (Sec. 1, Tab. 1 & 2).\n\n## 2. We will improve our introduction to Section 3, explaining the integration of the loss and architecture. (W2)\nWe will now start Section 3 (i.e. replace L109-114) with these paragraphs instead:\n\n> We follow the common idea from neuroscience [6-8] that cortical networks learn to interpret novel information by learning new top-down modulations, and propose that consolidation of these modulations is a crucial component of learning in biological brains. This motivates our task-modulated contrastive learning (TMCL) algorithm as the machine learning analogue of this consolidation, tackling continual representation learning. We consider the parameters of conventional machine learning models as *task-agnostic* feedforward weights $\\\\mathbf{W}$. On top of these weights, we introduce per-task affine transformation parameters as *task-specific* modulations $\\\\mathbf{m}\\_t$, as an analogue of biological top-down modulations (Eq. 2 in the curr. manuscript). We denote the modulated network as $f(\\\\mathbf{x}|\\\\mathbf{W}, \\\\mathbf{m})$ with feedforward weights $\\\\mathbf{W}$ and modulations $\\\\mathbf{m}$, while $f(\\\\mathbf{x}|\\\\mathbf{W}, \\\\emptyset)$ represents the unmodulated network (i.e. where the modulations are identity operations).\n\n> In TMCL, the overall objective is to arrive at an unmodulated representation space where all classes $c \\\\in \\\\mathcal{C}$ in the dataset $\\\\mathcal{D}$ have compact representations clustered around mutually orthogonal class centers, i.e. $$\n\\\\gamma^c\\\\bot\\\\gamma^{c^{\\\\prime}},\\\\forall c,c^{\\\\prime}\\\\in\\\\mathcal{C}, c \\neq c^{\\\\prime},(1)$$ \nwith $\\\\gamma^c = \\\\mathbb{E}\\_{\\\\mathbf{x} \\\\in \\\\mathbf{X}^{(c)}}[f(\\\\mathbf{x} \\\\vert W, \\\\emptyset)]$, where $\\\\mathbf{X}^{(c)}$ is the set of samples from class $c$. Because we assume a continual learning setting, where we do not have all class samples at our disposal, we do not optimize for (1) directly. Rather, we achieve this by breaking the optimisation procedure down into two distinct learning objectives. The first objective is to **orthogonalize** any given class $c$ from the others in a *modulated* representation space, i.e. we learn a modulation $\\\\mathbf{m}^{c}$ so that the class center of class $c$ becomes orthogonal to all other classes in the modulated space: \n$$\\\\gamma^{c}\\_{\\\\mathbf{m}^{c}} \\\\bot \\\\{ \\\\gamma^{c^{\\\\prime}}\\_{\\\\mathbf{m}^{c}} : c^{\\\\prime} \\\\in \\\\mathcal{C} \\\\setminus \\\\{c\\\\} \\\\}, (2)$$ \nwhere $\\\\gamma^c\\_{\\\\mathbf{m}}$ denotes the representation of the class center under modulation $\\\\mathbf{m}$, i.e. $\\\\gamma^c\\_{\\\\mathbf{m}} = \\\\mathbb{E}\\_{\\\\mathbf{x} \\\\in \\\\mathbf{X}^{(c)}}(f(\\\\mathbf{x} \\\\vert W, \\\\mathbf{m}))$, via Eq. [3 in the curr. manuscript]. The second objective (**consolidation**) is entirely unsupervised and trains network weights to become *modulation-invariant*, so that $$\n\\\\gamma^c = \\\\gamma^c\\_{m^{c^{\\\\prime}}} \\\\forall c^{\\\\prime} \\\\in \\\\mathcal{C}, (3)$$ \nvia [Eq. 5, Tab.1 in the curr. manuscript].\nIt can be seen that a representation space for which *both* (2) and (3) hold, implies (1).\n\n> In our continual setting, which we adapt from Fini et al. [1], training is partitioned into $s\\\\in{1,\\\\ldots, S}$ sessions, and therefore (1) can only be achieved approximately. In each session $s$, we only observe unlabeled samples $x\\\\in\\\\mathcal{D}^{(s)}\\\\subset\\\\mathcal{D}$ belonging to the session-specific partition of classes $\\\\mathcal{C}^{(s)}\\\\subset \\\\mathcal{C}$. Additionally, few labeled samples $(x, y)\\\\in\\\\mathcal{D}^{(s)}\\_\\\\text{sup}\\\\subset \\\\mathcal{D}^{(s)}$ are made available to (3). As a consequence, during each session (Fig. 2), we learn objective (2) restricted to $\\\\mathcal{D}^{(s)}\\_\\\\text{sup}$ in a first phase, and then learn objective (3) using unlabeled samples from $\\\\mathcal{D}^{(s)}$. We explain the implementation of both phases in detail below.\n\nThe combination of `VI + MI` will be explicitly marked as \"TMCL\" in the tables.\n\n## 3. The batch setting is chosen for comparability to literature. (W3)\nIn literature, continual self-supervised learning (SSL) is performed entirely in a batched setting. In fact, while we believe that SSL - as an implementation of predictive coding - is a biologically plausible mechanism, online self-supervised learning is an active area of research and out of scope for this work. Orthogonalization could be implemented in an online fashion, and remains to be explored in future work. We note finally that TMCL could be adapted straightforwardly to a setting where new classes appear asynchronously in the data stream, by replacing the negatives in the OPL with randomly draw samples from the unsupervised stream.\n\n## 4. Questions\n### On the terminology of \"modulation\" and \"top-down\". (Q1)\nWith \"top-down\", we refer to the origin of the modulations in the biological setting, i.e. we expect modulations to originate from higher-order cortical areas, shaping neuronal behaviour based on high-level information (such as the current task). Indeed, we refer to the element-wise multiplicative and additive parameters as \"modulations\". These are separate from the task-agnostic weights, and in contrast to the weights, they are per-task.\n\nRegarding BatchNorm, we intended to express that **per-task** BatchNorms would be equivalent, resulting in per-task elementwise trainable shifts $\\\\gamma\\_t$ and biases $\\\\beta\\_t$ (for all tasks $t$).\n\n### Modulations can be considered a parameter-efficient fine-tuning technique, but our main contribution is the *consolidation* of task-specific modulations. (Q2)\nWe aim to clarify that **the novelty of our method is** not the idea of task-specific modulations for model adaptation (for which similar methods have been explored in literature), but **the consolidation of task-specific modulations**. In fact, TMCL would allow for the consolidation of \"1-vs-all\" orthogonalizing LoRA adapters, which is out of scope for our biologically grounded paradigm.\n\n### We provide prior and new results with replay-based methods. (Q3)\nOur work builds upon continual representation learning literature, in particular CaSSLe [1], which investigate the replay-based methods ER [2] and DER [3] in a completely unsupervised setup (i.e. `VI + ER`, `VI + DER`), and find that these methods underperform `SI (CaSSLe)`.\nWe obtain new results with CLS-ER [4] - a biologically-inspired method we compare to in our related work section - which underperforms our method `VI + MI` given all labels and deteriorates significantly when only provided 10% of the labels (cf. Tab. 4, rebuttal to `4K39`).\n\n### Further questions\n> Are the modulations and weights trained in completely independent phases? **(Q4)**\n\nYes, they are (as outlined in Figure 2 in the manuscript) made more clear in the new start of section 3).\n\n> If there are two phases, how is the data split between the phases? How can you train with the contrastive loss in the \"task-learning\" phase if it seems to require orthogonalization between classes, when the data is mostly unlabelled? **(Q5)**\n\nThis is now explained in the new start of section 3.\n\n> [...] I wonder if this representation may impede effective generalization in cases where it is not needed, such as in a single task-setting? [...]\n\nWe indeed expect modulation invariance (`MI`) to hurt generalization in a single task setting, or if the tasks do not cover the problem space sufficiently. Nevertheless, we empirically find that CIFAR-100 one-vs-all tasks cover the problem space sufficiently well to achieve strong generalization (Table 3 in the submitted manuscript).\n\n> Line 172: what is the rationale for using the average of the views here?\n\nWhile the idea of taking an average originates from a known multi-view version of Barlow Twins [5], the idea is compatible with a simple idea from vector symbolic architectures (VSA):\nFor sample $\\\\mathbf{x}$ and class $c$, $f(\\\\mathbf{x}|\\\\mathbf{W}, \\\\mathbf{m}\\_c)$ describes a feature vector - e.g. \"x is dolphin\", \"x is not cat\" and \"x is not dog\". The objective is to minimize the distance of the unmodulated representation $\\\\mathbf{v}=f(\\\\mathbf{x}|\\\\mathbf{W}, \\\\emptyset)$ towards all of these \"feature vectors\", for which the optimal solution is to pull $\\\\mathbf{v}$ towards the centroid of these feature vectors.\n\n> Table 1: it is comparatively less clear what the set of modulations are, how they are sampled, and why there are an equal number of modulations [...] and augmentations [...]\n\nFor each view, we draw a separate augmentation (except for `SI`, following literature) as well as a separate modulation (`MI`).\n\n> Some theoretical justification for this approach would help clarify what the contribution of this method is beyond the presented motivation.\n\nWe refer to the theoretical considerations we formulated in the rebuttal to `vrza`, as well as in the new start of section 3.\n\n### Minor Comments\n- We will streamline the usage of session and task to make sure that \"task\" refers to the tasks solved by our one-vs-all modulations.\n- $\\\\mathcal{L}\\_\\\\text{CL}=\\\\mathcal{L}\\_\\\\text{MV-BT}$ for all methods except `SI`, for which we use $\\\\mathcal{L}\\_\\\\text{CL}=\\\\mathcal{L}\\_\\\\text{BT}$.\n\n----\n### References (cf. manuscript)\n[1] Fini et al. 2022\n[2] Robins. 1995\n[3] Buzzega et al. 2020\n[4] Arani et al. 2022\n[5] Ghosh et al. 2024\n[6] Iyer et al. 2022\n[7] Wybo et al. 2023\n[8] Gilbert et al. 2013"}, {"review_id": "XfsH5ffjhT", "reviewer": "Reviewer_vrza", "summary": "This paper tackles the problem of catastrophic forgetting, especially in cases where models need to learn from both unsupervised and supervised data. To solve this problem, this paper introduces task-modulated contrastive-learning (TMCL). TMCL has a feedforward backbone that used contrastive loss for unsupervised learning. TMCL also has a network that is trained to learn modulations to implement supervised learning of object identities (while the feedfoward backbone is frozen). To learn modulation-invariant representations, the feedforward network is subsequently trained to with a contrastive loss to bring together representations of differently modulated views of the same image and push those representations away from the unmodulated representation of another image. They test TMCL against state-of-the-art alternative models, and their model performs in the same ballpark (often better than the other models, but generally within a few percentage points).", "questions": "To make this easier to follow and respond to during the rebuttal, I’ve ordered my questions and suggestions to correspond with the points raised in the Weaknesses section above.\n\n1. Model clarity\n\n     a) I'd like to see the model explained fully and more clearly earlier in the paper. This would help readers have context around Figures 1 & 2 and even Related Work. I realize that it is rare to put a model description before Related Work, but page 5 is quite late to fully explain the model, and readers might be better equipped to put the Related Work in context if they already understand the model.\n     b) I suggest revising Figure 2 to make it easier for readers to follow. \n     c) What is the architecture of the modulated network?\n\n2. Orthogonal representations of class identities\n\n     Why does contrastive training of modulated views lead to an unmodulated representation space with orthogonal class identities? Can the authors walk this logic out in the paper more clearly?\n\n3. Marginal performance gains\n\n     Do the authors believe that their model is promising because of superior performance or because TMCL has novel biological inspiration and still performs comparably to state-of-the-art models? If the latter, I suggest explicitly making this argument in-text.\n\n4. Confusing ablation results\n\n     a) Why do untrained/random modulations perform nearly as well as trained modulations? The authors note that \"This suggests that predicting representations based on a slightly perturbed model state already provides a regulatory effect.\" However, this does not sufficiently clarify why performance is so close to non-ablated VI+MI performance. \n     b) The authors claim \"Removing the stop-gradient while keeping the modulations frozen results in a moderate performance degradation,\" but that \"moderate\" degradation is only 0.7%. If the stop-gradient makes so little difference, does this call into question the necessity of a \"hypothesized cortical predictive coding pathway\"? \n     c) Why is performance higher without pred. **and** stop grad than just without pred.?\n\nOne last question unrelated to the weaknesses: The paper claims that \"information [arriving] at spatially segregated loci on sensory neurons [suggests] distinct roles in shaping the neuronal input-output relation\" and distinct plasticity processes (page 3). Can the authors clarify why information arriving at different loci should imply different plasticity processes?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- This paper tackles catastrophic forgetting, a long-standing and important issue in machine learning. Their approach to understanding catastrophic forgetting (noting that ecological learning is generally unsupervised with limited access to supervised labels) is also a refreshing approach.\n- I appreciate the biological grounding of the model.  The authors draw inspiration from complementary learning systems theory, mapping fast learning to task-driven modulation and slow learning to unsupervised consolidation. This biological framing is conceptually interesting and helps motivate the model architecture. Ultimately, I don't think this *needs* to be the best performing model to be interesting. It is simply interesting that this different approach can still compete with other state-of-the-art models.\n- I generally think that contrastive learning is a very promising approach. Contrastive learning has lost a little popularity as MAE approaches have gained popularity, but I'm still quite enthusiastic about the contrastive learning framework. This paper adapts contrastive learning in a novel way.\n- TMCL performs comparably (and sometimes slightly better) than existing state-of-the-art models across several tasks.\n\n## Weaknesses\n1. One of my main concerns is clarity in terms of the model description. The high-level description of the model is difficult to follow, particularly early in the paper. Key details about how the system works only become clear around page 5 (lines 156–166), which might be too late to engage the reader effectively. Figure 2, while clearly intended to clarify the architecture, is difficult to parse. The relationship between the top and bottom panels is unclear, and labels like “Learning Pathway: modulations” do not meaningfully aid interpretation. I also do not have a confident grasp on what the architecture of the modulated network looks like.\n2. The logic behind how contrastive training of modulated views leads to an unmodulated representation space with orthogonal class identities isn’t clearly justified. This seems to be an important theoretical claim, but it’s not adequately walked through. (The authors claim that this is true \"intuitively,\" line 158).\n3. Tables 2 & 3 suggest the model is performing similarly to others, with modest gains (~1–2%). This makes the paper feel like another iteration in the pattern of marginal improvement papers, rather than a conceptual leap forward.\n4. The ablation results raise some questions and concerns for me. While it’s great to see ablations included, some of the results are unintuitive or potentially undermining. For example, untrained/random modulations perform nearly as well as trained ones. Also, performance is higher without pred. AND stop grad than just without pred. These results are not well explained and leave open questions about what components are actually contributing to the model’s success.", "was_revised": true, "final_justification": "The authors clearly addressed the weaknesses from my review (see my last comment to the authors).\n\nI also read through the other reviewers' comments to see if there were other critical issues, and my general summary isL\n* Modulation scaling - To me, this is an interesting area for future research, which the authors appear to be working on based on their responses.\n* CIFAR-100 - The authors added ImageNet experiments, which fully resolves this concern\n* Comparison to other models - Personally, I don't see why having some biologically inspired models makes other biologically inspired models less interesting. This model strikes me as unique relative to models and baselines.\n* Choices in the model that are not biologically inspired (batch setting) - I agree with the reviewer that this is a minor concern. This strikes me as an interesting future follow up idea. But we stifle model innovation if we expect papers initially introducing models to replace every ML technique with biological implementations. I think this contribution is sufficient without a streaming setting.", "rebuttal": "We would like to thank the reviewer for their valuable feedback on our work. We are encouraged by the reviewer's appreciation of the biological grounding and the novel conceptual ideas of our manuscript. We improve the clarity of our model description and the intuition of the consolidation phase (W1, W2), and also provide new, more significant experimental results and ablations (W3, W4).\n\n### 1. TMCL is primarily a machine learning analogue to our hypothesized cortical learning mechanism, but also provides significant improvements in learning *representations* continually. (W3/Q3)\nOur primary goal for this work was to achieve a conceptual algorithm that explains cortical learning. In particular, our algorithm aims to provide a computational framework that makes sense of the general cortical layout with top-down connections arriving distally on the dendritic trees of the neurons, representing modulations of the neural activity. Complementarily, we still aimed to demonstrate strong continual learning performance. We will make this hierarchy of objectives more clear in the introduction of the camera-ready version.\n\n### 2. We will improve the high-level description of the model at the beginning of Section 3. (W1/Q1)\nWe refer to the rebuttal to reviewer `XmrC` (Section 2), in which we provide the significantly improved description.\n\n### 3. Performance improvements are significant with weaker ResNet baseline, and on the more complex ImageNet-100 dataset. (W3/Q3)\nOn ImageNet-100, TMCL outperforms all other methods in the semi-supervised setting, improving over the `VI` baseline by +3.8pp (10% labels) and +2.3pp (1% labels). In the fully supervised setting, we observe TMCL improves over the `VI` baseline by +4.8pp on ImageNet-100 (`fD14`), and by +4.7pp on CIFAR-100 using ResNet-18 (Tab. 5 in the curr. manuscript).\nWe refer to the rebuttals to `4K39` (Section 3) and `fD14` (Section 1, Tables 1 & 2).\n\n### 4. We improve the explanation on how consolidating orthogonalized modulations introduces orthogonalization in the unmodulated representation space. (W1,2/Q1,2)\nWe appreciate the feedback about the clarity of the exposition in our manuscript. Thanks to the reviewers' comments, we now realise that we should have been more clear in our description of the principles underlying our TMCL algorithm. We again refer to the rebuttal to reviewer `XmrC` (Section 2) for the improved explanation.\n\nHere, to walk out the logic in further detail, let us explain the idea in a non-incremental fashion with four classes $A, B, C, D$. For conceptual clarity, we focus the explanation here on the class centers of these respective classes.\n\n 1. In the orthogonalization phase of TMCL, we train modulations $m\\_A$ to achieve $A\\\\vert\\_{m\\_A} \\\\bot \\\\{B\\\\vert\\_{m\\_A},C\\\\vert\\_{m\\_A},D\\\\vert\\_{m\\_A}\\\\}$, $m\\_B$ to achieve $B\\\\vert\\_{m\\_B} \\\\bot \\\\{A\\\\vert\\_{m\\_B}, C\\\\vert\\_{m\\_B}, D\\\\vert\\_{m\\_B}\\\\}$, $m\\_C$ to achieve $C\\\\vert\\_{m\\_C} \\\\bot \\\\{A\\\\vert\\_{m\\_C}, B\\\\vert\\_{m\\_C}, D\\\\vert\\_{m\\_C}\\\\}$, and $m\\_D$ to achieve $D\\\\vert\\_{m\\_D} \\\\bot \\\\{A\\\\vert\\_{m\\_D}, B\\\\vert\\_{m\\_D}, C\\\\vert\\_{m\\_D}\\\\}$. Here, $A\\\\vert\\_{m\\_A}$ denotes representational vector of the class center of class A under modulation $m\\_A$, and similar for all others.\n 2. In the consolidation phase, our goal is to arrive at a representation space where $A\\\\vert\\_{\\\\emptyset} \\\\bot B\\\\vert\\_{\\\\emptyset} \\\\wedge A\\\\vert\\_{\\\\emptyset} \\\\bot C\\\\vert\\_{\\\\emptyset} \\\\wedge A\\\\vert\\_{\\\\emptyset} \\\\bot D\\\\vert\\_{\\\\emptyset} \\\\wedge B\\\\vert\\_{\\\\emptyset} \\\\bot C\\\\vert\\_{\\\\emptyset} \\\\wedge B\\\\vert\\_{\\\\emptyset} \\\\bot D\\\\vert\\_{\\\\emptyset} \\\\wedge C\\\\vert\\_{\\\\emptyset} \\\\bot D\\\\vert\\_{\\\\emptyset}$ (where $A\\\\vert\\_{\\\\emptyset}, B\\\\vert\\_{\\\\emptyset}, C\\\\vert\\_{\\\\emptyset}$ and $D\\\\vert\\_{\\\\emptyset}$ denote the class centers of $A,B,C,D$ in the unmodulated network). It can be seen that this will be the case, if we simultaneously achieve the orthogonality relations of point 1 *and* $A\\\\vert\\_{\\\\emptyset} = A\\\\vert\\_{m\\_A} = A\\\\vert\\_{m\\_B}  = A\\\\vert\\_{m\\_C} = A\\\\vert\\_{m\\_D}$ (where $A\\\\vert\\_{m\\_{A,B,C,D}}$ denotes the class center of class $A$ respectively under modulations $m\\_A$, $m\\_B$, $m\\_C$, and $m\\_D$) and similar for classes B, C, D. The contrastive objective maximizes similarity between a given data sample without modulation and under modulations  $m\\_A$, $m\\_B$, $m\\_C$, and $m\\_D$, and therefore implicitly drives the network to a state for which $A\\\\vert\\_{\\\\emptyset} = A\\\\vert\\_{m\\_A} = A\\\\vert\\_{m\\_B}  = A\\\\vert\\_{m\\_C} = A\\\\vert\\_{m\\_D}$ and similar for $B,C,D$.\n\nWe note that achieving the full orthogonality relation  $A\\\\vert\\_{\\\\emptyset} \\\\bot B\\\\vert\\_{\\\\emptyset} \\\\wedge A\\\\vert\\_{\\\\emptyset} \\\\bot C\\\\vert\\_{\\\\emptyset} \\\\wedge A\\\\vert\\_{\\\\emptyset} \\\\bot D\\\\vert\\_{\\\\emptyset} \\\\wedge B\\\\vert\\_{\\\\emptyset} \\\\bot C\\\\vert\\_{\\\\emptyset} \\\\wedge B\\\\vert\\_{\\\\emptyset} \\\\bot D\\\\vert\\_{\\\\emptyset} \\\\wedge C\\\\vert\\_{\\\\emptyset} \\\\bot D\\\\vert\\_{\\\\emptyset}$ rigorously would require iterating steps 1 and 2, and under certain conditions we expect a convergence proof can be constructed for this.  However, we did not seek such an iterating implementation, as we focused on the continual learning setting where we identified modulation learning (step 1) with a single phase of fast learning that occurs whenever a new class is observed, and step 2 with a slower consolidation phase. Under these conditions, the full orthogonality relation can not be expected to be achieved rigorously, but as is demonstrated by our CDNV results, our TMCL algorithm still leads to a representation space where the clustering of the individual classes is improved.\n\n### 5. Orthogonalization ablations are significant on ImageNet-100. (W4/Q4)\nWe believe that the combination of a stronger ConViT baseline as well as the simple CIFAR-100 dataset explain the small differences in ablation performances.\nTherefore, **we have now also performed the orthogonalization ablations on ImageNet-100, and observe much stronger differences under linear evaluation (Table 5)**.\nWe will include these novel results in the camera-ready version to demonstrate the effectiveness of TMCL.\n\n### 6. Interpretation of the ablation results. (W4/Q4)\nAs for the specific points, regarding *(a)* we believe that maximising similarity over noise modulations enhances robustness of the representations, therefore leading to a higher performance than pure VI, in the same way as adding noise as an image augmentation also leads to higher performance (see e.g. [1]). \nRegarding *(b)*, we note that we consider the predictor + stop-grad as the more biologically plausible choice, as in a streaming setting the predictor could be interpreted as providing a representation originating from sensory input from the previous time point, allowing it to be compared to sensory input from the current time-point. Without stop-grad, synaptic plasticity would have to proceed based on two loss values at the same time, whereas having a neural implementation with only one loss value is already challenging (see e.g. [2]). \nRegarding *(c)*, at this point we only have these empirical observations, any statement we make would be speculation. \n\n### 7. Distinct plasticity based on spatial location. (Q5)\nBroadly speaking, synaptic plasticity is governed by calcium influx, which happens when N-methyl-D-aspartate (NMDA) receptor channels in excitatory synapses are opened. These receptor channels need strong depolarization to open, which, at the soma, is mostly provided by action potentials, leading to classic spike-timing dependent plasticity. More distally in the dendritic tree, however, such depolarization is provided by local inputs, thus leading to very different plasticity dynamics. These effects are well documented in the neuroscience literature, and we suggest Kampa et al. [3] as a good starting point.\n\n---\n\n**Table 5:** Ablations on ImageNet-100, averaged over 3 seeds\n\n| Method                            | kNN  | linear |\n| --------------------------------- | ---- | ------ |\n| VI                                | 50.5 | 59.7   |\n| **VI + MI (ours, TMCL)**                | 56.0 | 64.5   |\n| &nbsp;&nbsp;Untrained modulations | 50.7 | 60.6   |\n| &nbsp;&nbsp;Random modulations    | 50.5 | 60.9   |\n| &nbsp;&nbsp;OPL -> BCE            | 56.1 | 63.8   |\n\n\n#### References\n[1] Chen, et al., A simple framework for contrastive learning of visual representations. ICML 2020.\n[2] Sacramento et al., Dendritic cortical microcircuits approximate the backpropagation algorithm. NeurIPS 2018.\n[3] Kampa et al., Dendritic mechanisms controlling spike-timing-dependent synaptic plasticity. Trends in Neurosciences (10.1016/j.tins.2007.06.010)"}]

Author rebuttal:

Meta-review: The paper introduces Task-Modulated Contrastive Learning (TMCL), a biologically inspired approach to semi-supervised continual learning that integrates sparse supervisory signals into a contrastive learning framework. By leveraging modulations learned from limited labeled examples, TMCL aims to learn modulation-invariant representations, mitigating catastrophic forgetting while learning new concepts in a task-agnostic way. TMCL is evaluated extensively on CIFAR-100 (and ImageNet-100 in rebuttal) and shows improvements over baseline methods in many scenarios.

There was significant variance in initial reviewer opinions of this submission. In particular, Reviewer 4K39 questioned the novelty and significance of the work to the broader continual learning community. Through a sequence of vigorous back-and-forth exchanges between the authors and Reviewer 4K39, the reviewer relaxed their overall objection to the work, and the authors convincingly argued that the approach is interesting both due to its biologically-inspired nature and the state-of-the-art results on the considered datasets. The novelty of the approach seems clear, based on the other reviewer comments regarding the implicit orthogonalization via invariance learning. As for the outstanding questions from Reviewer 4K39 regarding the significance of the contribution to the continual learning community, NeurIPS is a form for broader discussions on machine learning from diverse perspectives. This contribution is interesting it its combination of ideas into a novel, biologically grounded model for semi-supervised Continual Learning. It thus adds something to the discussion on continual learning, potentially bringing fresh ideas from the broader community to bear on its fundamental problems. The recommendation is thus to Accept.
Decision: Accept (poster)