Continual learning with the neural tangent ensemble
Authors: Ari S Benjamin
Christian-Gernot Pehle
Kyle Daruwalla
Venue: NeurIPS 2024 spotlight (2024)
Keywords: continual learning
catastrophic forgetting
Bayesian ensembles
Boosting and Ensemble Methods
mixture of experts
Primary area: online_learning
TL;DR: All network classifiers are ensembles; each edge provides a classifier. If you weigh them by their posterior probability you (almost) get SGD.

Abstract: A natural strategy for continual learning is to weigh a Bayesian ensemble of fixed functions. This suggests that if a (single) neural network could be interpreted as an ensemble, one could design effective algorithms that learn without forgetting. To realize this possibility, we observe that a neural network classifier with N parameters can be interpreted as a weighted ensemble of N classifiers, and that in the lazy regime limit these classifiers are fixed throughout learning. We call these classifiers the *neural tangent experts* and show they output valid probability distributions over the labels. We then derive the likelihood and posterior probability of each expert given past data. Surprisingly,  the posterior updates for these experts are equivalent to a scaled and projected form of stochastic gradient descent (SGD) over the network weights. Away from the lazy regime, networks can be seen as ensembles of adaptive experts which improve over time. These results offer a new interpretation of neural networks as Bayesian ensembles of experts, providing a principled framework for understanding and mitigating catastrophic forgetting in continual learning settings.

Continual learning with the neural tangent ensemble
Ari S. Benjamin
Christian Pehle
Kyle Daruwalla
Cold Spring Harbor Laboratory Cold Spring Harbor, NY 11724
{bn ami, ,eaeneeeeee,,aalallelll www.walllcshhlcccom
Abstract
A natural strategy for continual learning is weigh a Bayesian ensemble of fixed functions. This suggests that a (single) neural network could be interpreted as ensemble, could design effective algorithms that learn without forgetting. To realize this possibility, we observe that a neural network classifier with parameters can be interpreted in the lazy regime limit these classifiers are fixed throughout learning. We call these classifiers the neural tangent experts and show they output valid probability distributions over the labels. We then derive the likelihood and posterior probability of each expert given past data. Surprisingly. the posterior updates for these experts are equivalent to scaled and projected form of stochastic gradient descent (SGD) over the network weights. Away from the lazy regime, networks can be seen as ensembles of adaptive experts which improve over time. These results offer a new interpretation of neural networks as Bayesian ensembles of experts, providing a principled framework for understanding and mitigating catastrophic forgetting in continual learning settings.
weighted ensemble of N classifiers, and that
Introduction
Neural networks often forget previous knowledge when trained with gradient descent. In contrast, animals learn from sequential experiences, suggesting that true 'lifelong learners' use different strategies for learning (25)-
One strategy to learns without forgetting is to update the posterior distribution over a set of fixed probabilistic models [8]. This includes any fully Bayesian model, such as Bayesian linear regression. The fundamental reason why these algorithms do not forget information is because the posterior over models is invariant to data sequence. Given two permutations of the data, the posterior will be the same. This property of posteriors has inspired many strategies to reduce forgetting by approximating the posterior distribution these approximations introduce many new parameters and considerable memory overhead. In general, estimating the full posterior distribution over high- sigh--imensional networks is prohibitive.
neural networks [22 24 I1 28 26 37]. However,
Here, we shift our perspective and instead interpret single neural network as an ensemble of many experts. This allows tracking a posterior (over experts, instead of networks) without introducing any memory overhead besides the network itself.
This motivates our main result, which we note is generally applicable outside of continual learning. More specifically, we show that neural network classifiers perturbed by small vector in param- eter space can be described as weighted ensemble of valid classifiers outputting probability distribution over labels. We call this the Neural Tangent Ensemble (NTE), Inspired by the Neural Tangent Kernel, this result depends on first-order Taylor expansion around a seed point [19]. As consequence, operates as an ensemble of fixed classifiers in the NTK limit of infinite width.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this framework, learning is framed as Bayesian posterior updating rather than optimization. These two approaches might be expected be quite different, as a posterior update is multiplicative whereas gradient- based optimization is additive. Surprisingly, however, we find that the NTE posterior update rule is approximately stochastic gradient descent (SGD) on the network with batch size 1, thus shedding new light on the dynamics of neural network optimization.
Our primary contributions are:
We introduce the Neural Tangent Ensemble (NTE), novel formulation that interprets networks as ensembles of classifiers, with each parameter contributing one classifier.
We derive the posterior update rule for the NTE for networks in the lazy regime, in which experts are fixed, and show that it is equivalent to single-example stochastic gradient descent (SGD) without momentum, projected to the probability simplex.
This justifies the empirical finding that SGD with no momentum forgets much less than standard optimizer settings.
We demonstrate that catastrophic forgetting in neural networks is associated with the transi- tion from the lazy to the rich regime.
2 Motivation: Ensembles are natural continual learners
To demonstrate why Bayesian ensembles make good continual learn- ers, consider function f(x) that is an ensemble of many experts fi(x) (Fig. 1). We will consider what it takes modify this ensem- ble so that it performs well on two tasks A and B.
simple strategy for continual learning is prune away experts. Let SA be the set of functions that are good (and equally good) for task A. A good ensemble can be constructed by sampling from SA:
SB
fx
Σ f((x). JESAA
Figure 1: High-level intuition for model averaging and con- tinual learning. Pruning the set of functions f. to those good for task A. followed by further pruning for tasks B and C, will result in a set of f; still good on A.
Given a subsequent task B, new ensemble can be constructed on the fly by continuing to prune away the experts in fa) that perform poorly on task B. The remaining ensemble is still composed of experts from SA (assuming that the set intersection is not zero).
In contrast to many continual learning strategies for neural networks, this does not require replay, task boundaries, or any additional mem-
ory dedicated to preserving old task performance.
2.1 Belief updating generalizes set intersections
In real ensembles, experts do not perform equally well. This justifies
weighing each expert in the ensemble with weights ps which are chosen such that Σ pt =
fa(x) = P.f(s)). f
(1)
This is particularly convenient if the experts encode the probability or belief about event, fz(z) plyl,, ft). In this case, one can weigh each function by its posterior probability given previous data:
ply|,, D) = Σ P((DD) p(yla,
(2)
J.EF
This is the optimal weighing strategy when the experts can be assumed to be independent 445
It is useful to contrast the ensemble in Eq. 2 with linear regression using feature map, f(x) E, (x)) as might be observed kernel regression. In an ensemble the weights w; are strictly positive, whereas weights in regression may switch sign arbitrarily.

---

2.2
The posterior is invariant to data ordering
The property of Bayesian ensembles that motivates this paper is that the posterior probability cach expert is invariant to the order in which data in seen. This is because, like set intersections, single-task posteriors multiply to form the multi-task posterior:
p(J[AAB) & P(AAAA)((S11)))
This property is restated more formally in the following Lemma:
Lemma 1. Invariance data ordering in Bayesian Ensembles. Let F fi...,,N be set of fixed experts, W = Wi...., wN be their weights, and D Di...., Dr be a sequence of datasets from T tasks. Then, for any permutation π of the indices .... T, p p(f (D) p(fil D1) Dxc)...>...,
Proof. By Bayes' rule, p(jiDD) x p(f.) T (D...)). The right-hand side is a product of terms,
one for each dataset. Since multiplication is commutative, I ((D.lf)) = II (() fi) for any permutation æ. Therefore, p(fi Dı,... Dr) = p(fa Pa(1):-.. Da(T)).
Thus, there is no catastrophic problem probabilistic classifiers. This motivates assessing under what conditions neural networks approach this setting.
problem for models which are ensembles of fixed, independent
3 The Neural Tangent Ensemble
How might neural network be described as ensemble? One simple strategy would be to take the last network layer as a set of functions, and then to choose the output weights according to their relative performance. However, this is is expensive strategy construct a relatively small set of classifiers, and it does not specify how earlier weights might change.
Here, employ first-order Taylor expansion to show that neural networks are (approximately) large ensembles over N component functions, one for each edge in the network. We will examine neural network p(y (a, W(() with parameters W(+) whose output represents the probability or confidence of label y given input x. We can describe this output with linearization around very nearby seed point ("), Note that we use the notation W0()) and (t))) for consistency and in general (0()) need not be the initialization or on the optimization trajectory at all.
plulæ, w(t)) so plu,z, p(ulx, + Σ Aw; 8p(u|z: x W
Ow (0)
At first glance it does not appear that this Taylor expansion is an ensemble. There seem to be no true classifiers: the gradients are not probabilities over classes y, being neither nonnegative, bounded, nor normalized to I across the output labels. Nor are there true weights, as Aw, is also not nonnegative. However, both of these criteria can be met with some rearrangements and with the assumption that the loss is sufficiently smooth with respect to its parameters. This leads to our main result:
Theorem 2. Suppose ply|z, W()) is a neural network for which the log ligeeiiko is L Lipschitz continuous in its parameters, i.e. all gradients of the loss are bounded by a constant L. Let ( then be perturbed by a AW with ||W]]] z. If the perturbation is sufficiently small (with zL < 1), then the network can be described as an ensemble of a set of N classifiers {ply)æ, zz fo) each with
weight [Ami] plus higher-order contributions which vanish for small z:
N
Awp
ply/2, w(())=
Σ
p(ylz, f.) + O(IAW13)
z
weightx
3

---

Each classifier plyla, fi), which we call the neural tangent expert, outputs probability distribution over labels y:
(1 (1+ z sign( (Aws)
plyl,, ft) p(yl2r, W(o)
(0) log p(yle, W(l))
dw.
The proof is postponed to Appendix 8.1 Informally, it relies two simple rearrangements. splitting the weights into sign and magnitude Aw; A (arelsiignnuuu and bringing the zeroth order term inside the first-order sum. This results in a sum over a term which, surprisingly, sums to over the output labels and is weighted by term that sums to I over experts, meeting the criteria of ensemble.
This simple reformulation invites a change in perspective about the role of each parameter in a deep neural network. Each parameter contributes separate classifier. The distributed architecture and connected paths of the network matter, but they explicitly contribute through the gradients alone.
In the literature ensembles, common focus is to examine the quality and diversity of the experts separately. By the bias/variance decomposition, both aspects enter in the generalization error [38 47 Here, it is clear that all experts share factor that is the overall quality of the center of the Taylor expansion, p W( What distinguishes experts from one another is the diversity of network gradients.
3.1 Experts are fixed in the lazy regime
This paper is motivated by the fact that Bayesian ensembles of fixed experts do not forget past data when learning by posterior updating. Under what conditions is the Neural Tangent Ensemble composed of fixed functions?
The answer to this question follows directly from the literature on the Neural Tangent Kernel (NTK) and lazy regime networks [19 77. If the network is in the 'lazy' regime, then the Jacobian of the network does not change during gradient descent learning and the linearization remains valid. This occurs in the limit of infinite width for MLPs for certain initializations [19]. (Output scaling also controls laziness 77. and is a necessary when using softmax nonlinearities even the infinite width 29
As consequence, the experts in the NTE interpretation are fixed functions in the lazy regime.
3.2
Learning ensemble weights
If network is secretly ensemble, how should it learn from new data? The natural next step is to convert the NTE into Bayesian ensemble. In a Bayesian ensemble, the weight each function its posterior probability given past data:
Awa
p(f(D) = p(D(S))(S))
(5)
E,P(P(s)P())
This be as the step in generalized EM algorithm 45 In the following section, will describe how to calculate this posterior probability with an online learning algorithm. For the moment, we assume the experts are fixed functions (i.e. the network is lazy).
3.2.1 The data likelihood
Lemma 3. For IID data D {zk, yk Hh= the likelihood of the data under an expert can be written in terms of log-likee logliikll loss function e log p(uv w()) of the network at initialization:
p(DIf))
II examples
e -
c
(0)
(6)
ow!
(0)

---

Proof. Starting with the data likelihood,
p(D]fi)
II examples
p(y 2k fi)
(
II examples
PlUK|ZK, W(ol) + sign(Aw,)
p(us Ikk W(l))
(8)
Owl
plulaa w()o [ sign(Aw)
II examples
log logp(un [xs W(l))
(9)
Ow: (0)
Plugging in the definition of (00) yields the above expression.
3.2.2 The posterior probability: renormalization
After the data likelihoods are computed for each neural tangent expert, they must renormalized to obtain the posterior probabilities. In our case, we naturally have access to very large number of tangent experts and their likelihoods. Indeed, if the width is indeed taken to infinity, this there infinitely many neural tangent experts in single network. We propose to use the distribution of likelihoods in the current network as a Monte Carlo estimate the normalizing constant.
Texamples & e % (1 sign( Aw;) ((0) dw p(fi)
P(
(10)
Ex Leesmpplsss &
sign (wn)) aw
((0) to
p(fi)
This can be simplified by noting that each e % term will cancel; the product II. e appears in every additive term in the denominator. Assuming uniform prior p(fi), we then have:
4
Ih [ 1 sign( Aw))
4
P(f|D)
(11)
Ile (1- sign( (w:)) Bu
((0)
3.3 The posterior update (almost) stochastic gradient descent
We will now link this posterior expression with neural network update rule. Recall that in Theorem the normalized magnitude each perturbation interpreted as the posterior probability of the corresponding neural tangent expert.
Aw;
P(/(DD
This means the parameters standard belief updating, this involves a likelihood update followed by renormalization, Surprisingly, this multiplicative belief update rule yields an update which is very close SGD.
running cache of the posterior as new data is encountered. As in
first-order Taylor expansion around around (O) with perturbation ||W[]] = z. the posterior belief update given a new example is equivalent single-example stochastic gradient descent under cross pross-entropy loss objective, subject to the constraint that A = 2, and using a per-parameter learning rate of 2| z|Auo]l
Lemma 4. For any network that is deserdede
Proof. The proof is matter of writing out how the posterior changes with single example. Multi- plying by the likelihood of a new example, the unnormalized posterior updates as:
Aw? ((t)
Aw! (t-1)) (t-1)
1 sign( Aw? (t-1))
(0) (0)
(12)
Ow
5

---

This multiplicative update for the unnormalized weights can also be written an additive rule. Multi- plying by and by sign( (w;),
A Aw (t)) = Au;" (t-1)
z|Au; (t-
(13)
dw
(0)
&
One can add the initial parameters to either side to yield a rule in the space of network parameters:
W0
(t-1) wi
Aw
a
(14)
dw;
This is true (single-example) stochastic gradient descent projected in the LI diamond with learning rate z|Aw,]] Note that this does not allow averaging gradients across examples (a "batch size of 1" update) and that it uses the gradients at initialization (though see section 4.1
To complete the update, the parameters should then be renormalized such that >; I
An alternative normalization scheme is use gradient projection algorithm. Adding Langrage multiplier to Eq. 13 and solving for the y that ensures Σ; |Awy = yields update which keeps ||W]|| even without renormalization:
(1) w w
(t-1)
|
Ow! ())
(0)
avg,
|
|
Ow (0)
(())
&
(15)
Not only is the posterior update tractable, then, but it is sufficiently close to gradient descent that can be interpreted in standard optimization framework.
Although may seem that our result would depend on the idiosyncratic likelihood function of the NTE, this result is nevertheless similar to previous algorithms that have been proposed as ways to weigh many experts. At high level. our result appears similar the Multiplicative Weights algorithm described in M Another interpretation of this algorithm is as the approximated exponential gradient descent with positive and negative weights algorithm from [23] but applied the change in weights AW. There, it is derived by minimizing an arbitrary loss function under a constrained change in the relative entropy over ensemble weights to obtain the exponentiated gradient descent algorithm, which is then linearized with a Taylor expansion in the approximated version.
3.4 Summary of the NTE theory
The Neural Tangent Ensemble is interpretation of networks ensembles of neural langent experts. Updating the NTT of lazy networks as Bayesian ensemble creates perfect continual learner, in the sense that the multitask solution is guaranteed to be the same as the sequential task solution.
The posterior probability of each expert in the NTE is surprisingly tractable. Given a new example, the update rule is simple additive rule in the space of network parameters which can be interpreted as projected gradient descent scaled by the change in parameters since initialization.
Networks away from the lazy regime
In real finite-width networks, gradients change throughout learning. Since each weight's corre- sponding neural tangent expert changes, there is no guarantee that weights at time still reflect the cumulative likelihood of past data under that expert.
This phenomenon is clearly observed empirically by measuring how much experts change under NTE update rule function of network width. In Fig. 2 we measure the average change Jacobian from initialization after training on MNIST as a function of network width with the NTE rule described above. Experts change less in wider networks than in narrow networks.
in expert's s

---

Avg. Eadideen Distence between Experts
Jacoblan Alignmmnn with initialzazion
0.0012
Figure The average squared difference be- tween experts' columns of the Jacobian mea- sured at initialization and the end of training on MNIST with an 2-layer ReLU MLP and the NTE rule. Error bands indicate the standard deviation over 10 random seeds. As the width of the net- work increases, the average distance decreases, indicating the larger networks remain closer to the original linearization.
0.0010
0.0034
0.0002
10
102 10
10*
Witth
Another way this can be measured is by verifying that, in finite-widis networks, the posterior update rule using the gradients around initialization does not lead effective training. In Figure 3) we confirm that as the gradients lose correlation with the gradient at initialization, performance begins to rapidly degrade. This echoes the findings of 7 that linearized CNNs do not learn as effectively as their non-lazy counterparts. Thus, the NTE posterior update rule written above is only effective when the Jacobian is truly static.
Figure 3: a) Gradients of an MLP at time (t) rapidly lose correlation with the gradients at initialization. b) Training network with the NTE posterior update rule fails when gradients di- verge. Hyperparameters are reported in the Appendix.
Cozine zimliontty of graclentt
20
1308
1509
2003
250
536
1200
12:0
2000
2006
Stop
Stop
4.1 Rich-regime networks are ensembles of adaptive experts
To ensure the NTE formulation remains valid, one allow the seed point of the Taylor expansion (the 'initialization') to change throughout learning. This has an interesting interpretation. Namely, allows us view finite-width neural networks as ensembles of changing neural tangent experts.
Lemma 5. (informal) Let W(O) be the parameters of (finite-width) neural network. Choose nearby seed point (+) as Wit) + with fixed and ||||2 sufficiently small relative to the curvature such that the Jacobians of the log output probabilities of the perturbed and unperturbed networks are identical, J(W()) ((W()). The network can then be written as an ensemble of adaptive experts:
ply (()) p(y|x, p(ylx,11(10) 1 sign(e)))
log plyl,, W(t))
Owl
If is the uniform vector with values e Vn/N, the learning rale in the posterior update rule reduces to |e]]ale] = 1) and we recover stochastic gradient descent with mean- mean-centered gradients and learning rate m:
i
(1+1) W
w
aw
K
avg,
aw
(0)
( K
(16)
Rich-regime learning is thus akin to particle filter; each expert changes individually, but the prediction is the ensemble vote.
interesting feature of this lemma (gradient descent w) and the rule that decides how to weigh the experts in the ensemble (also gradient descent on w). This need not have been the case. As result, one can perform belief updating assuming fixed ensemble and end up improving each expert within it.
equivalence between the rule that improves each expert
7

---

4.2
The NTE rule with current gradients
Motivated by this result, we evaluated how well the NTE posterior update rule works when the gradients evaluated at initialization, 8 (0) are replaced with the gradients of the current network
Ow not
0 the
These converge in the infinite-width limit.
K
To obtain practical algorithm, we additionally modify the NTE update rule with two hyperparameters that control the learning rate. First, noting that in Eq. [4]act learning rate, replace it with tunable parameter Secondly, we introduce a regularization parameter 3 which keeps the network close to initialization as measured by the relative entropy of the change in parameters (see Appendix 8.2 for derivation). This constrains the amount of information contained in the weights [17].
Pseudocode for the resulting algorithm is in the Appendix [ over 3 and 77 on the Permuted MNIST task in Fig.
We also display the result of sweeps
Predictions and results
Our findings suggest several ways to reduce forgetting finite networks. First, networks closer to the lazy regime will better remember old tasks as long as the update rule is sufficiently similar to the NTE posterior update rule. Second, one should be able to reduce forgetting by ablating standard optimization methods like momentum and moving towards the NTE posterior update rule.
Below, verify these predictions the Permuted MNIST task with MLPs and the task- incremental CIFAR 100 with modern CNN architectures. In the Permuted MNIST task, an MLP with 10 output units tasked with repeatedly classifying MNIST, but in each task the pixels are shuffled with new static permutation. In dask-incremental CIFARI 100, a convolutional net with 100 output units sees only 10 classes each task. In the terminology of van de Ven et al, this is task insk-incremental task, whereas Permuted MNIST is domain-increment incremmntereal task [88
5.1 Momentum causes forgetting
Momentum is not appropriate in posterior update framework because it over-counts the likelihood of past data. Furthermore, it is a history-dependent factor. By contrast, posterior update rules are multiplicative and give identical results regardless of the order of data presentation.
Here, we report that any amount of momentum with SGD is harmful for remembering past tasks. To our knowledge, this has not been noted by previous empirical studies catastrophic forgetting
13 36 35
2] As can been seen in Fig. increasing momentum monotonically increases forgetting first task Permuted MNIST. Similar trends exist for ResNet18 and ConvNeXtTiny the
CIFARI00 task (see Fig. 8 30 Note that the momentum buffers were not reset between tasks; when they are reset, the momentum curve is nonmonotonie (see Fig. D. Although momentum assists single-task performance, any amount of momentum will lead to forgetting previous knowledge.
leaming
Single learning
100
90
1.00
First task peak accuracy
80
80
accuracy
Test acc. on task
after tasks
70
60-
95
60
50
40-
Taak-4
lask
10
20
Epoch
40
50
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Momentum
Momontum
Figure 4: Effect of momentum in SGD on the Permuted MNIST task for an MLP with layers and 1,000 hidden units. (middle) Test accuracy the first task the end of training 5 sequential tasks. (right) Final test accuracy on the first task before seeing the other tasks. Error bars represent standard deviations over seeds. See Appendix for further parameters.

---

5.2 Width improves remembering - but only for certain optimizers
As networks grow wider and (slowly) approach their infinite-win withh limit, they should remember better if one uses the appropriate posterior update rule over the Neural Tangent Ensemble.
Previous literature confirms that this is generally the case. In 440,. the authors report the benefits of scale are robust across architectures, tasks, and pretraining strategies, although they largely use SGD) with momentum learning benchmark algorithms such as EWC (22). Forgetting seems to be largely solved by scale.
0.9. In [3), the authors report similar results and investigate other continual
The reason for this in framework differs from the reason cited by both 35 40 which state that the gradients on different tasks will be more orthogonal in high dimensions, which reduces interference. Our interpretation is somewhat different and instead depends on the Jacobian of the network changes. We place no condition on gradient orthogonality between tasks. If the neural tangent experts indeed fixed, the NTE update rule will find the multitask solution.
If this the case, then wide networks should better remember only if the optimizer can interpreted as posterior update. In Fig. 5 we report that Adam's 's amnesia is not helped with increasing scale for the Permuted MNIST task. Although this could be for multiple reasons, we argue stems from a divergence from a valid interpretation as a posterior update.
5.3 The NTE posterior update rule using current gradients improves with scale
In Section 4.2 introduced modified version of NTE posterior update rule in which the Jacobian at initialization replaced with the current Jacobian. As networks get wider, this algorithm will converges the proper update rule due to the fact that the network Jacobian does not change in the lazy regime. This predicts that this rule will improve with scale. To test this, we trained an MI on Permuted MNIST and ConvNeXtTiny both single-task and multitask learning are greatly improved with width (Fig. aId Fig. [T]. We take this as empirical evidence that the proper NTE posterior update (with a static Jacobian) would work well in the infinite-width limit.
task-incremental CIFAR 100 with this approximate rule. We find that
100
90
Figure 5: Wider networks forget less, unless trained with Adam. See Alg. D All networks MLPs with ReLU nonlin- carities trained on 5 Per- muted MNIST tasks. Loss curves and further param- eters in Appendix. Er- ror bars represent standard deviations.
(p
80
Test acc. acc. on task 5
Test aCC. on task 1 after
96
2-layer
50-
92-
20
WIC 2.00. de0.al
Asam fq D0L 3.50-
to- 00
10
10a
10a
10t
107
100
10a
102
101
10"
10a
Number parameters
Number parameters
Related work
There is a long history of interpreting networks as ensembles. Networks with dropout, for example,
allow this interpretation 212 14] This is also closely related Mixture of Experts models in classic
18 20 and recent 43 work. These approaches explicitly encode the experts within the network, and unlike our work do not use a Taylor expansion establish the ensemble experts.
The idea of Bayesian ensemble over networks is also well-studied. Such ensembles can either be assembled empirically through sampling [51] 50], built via a Laplace approximation [3). or optimized 3.. Bayesian posteriors are also common players in theoretical works using methods from statistical physics and PAC-Bayes ensemble lazy learners
44 (7). Some treatments of infinite-width limits study the [6). While similar in spirit, these methods study groups of many networks rather than view single network an ensemble.
Finally, there related work that uses ideas from ensembles for continual learning. Many of these are in the category of methods that continually learn by training new modules for each task

---

49 42 52 39 21]. Most directly related to this current work are papers that take Bayesian
approach and track statistics about the approximated posterior over networks [22 T0 28
37
Many of these works in both categories require task boundaries. Furthermore, by introducing new modules or tracking statistics, these methods require additional memory to prevent forgetting.
6.1
Moving in directions of low curvature forget less
Our framework justifies the strategy of encouraging parameters to change mostly in directions of low curvature. Such regularization methods are already well-established and proven reduce forgetting Although not directly equivalent, this is also similar to Elastic Weight Consolidation, which penalizes by the Fisher Information matrix (an expected second-order derivative of the log- likelihood, rather than the likelihood)
24
31 41
[22]. Another proximal method is Synaptic Intelligence, which penalizes parameter changes proportional to their integrated gradients along the path, which in the special case of diagonal, quadratic loss functions, is equivalent the Hessian (3). Thus, second interpretation of why these methods work well (and improve with scale 355 that they ensure the langent experts in the NTE do not change much while learning.
7 Discussion
Here, described how networks in the lazy regime be seen ensembles of fixed classifiers. With this perspective, we proposed weighing each expert by its posterior probability to form Bayesian ensemble, and derived the update rule. This strategy of learning by posterior updating has the benefit that the order of data presentation does not matter sequential experience and interleaved experience lead identical ensembles.
The posterior update rule the Tangent Ensemble surprisingly similar to SGD on the model weights. However, it is interesting to note that this update rule is suboptimal. Posterior probabilities are the optimal ensemble weights only when the experts are independent 47 38 and well- specified [33 6]. This assumption is violated by the use of shared data, well as the fact that neural network architectures introduce dependencies between gradients. Although this does not affect the equivalence between the interleaved and sequential task performance (i.e. forgetting), this will reduce the performance networks trained with the NTE posterior update. This suggests avenues for improving SGD.
This suboptimality could be addressed in multiple ways. In the ensemble literature, there are many strategies to diversify the expert pool 4 such as repulsion (9. Different experts might also be trained on different data [46.. and one might even take boosting approach [5] It very likely that these approaches would yield neural networks that outperform standard networks trained by updating the
posterior distribution over tangent experts.
The ability in interpret single networks as ensembles opens many avenues for future research. These extend beyond continual learning; for example, one might be able obtain measure of uncertainty of the network output via the variance of the experts [2]. We are hopeful that this insight will lead to deep learning systems that are better understood their use expands within society.
Acknowledgements
The authors thank Peter Koo and Ben Cowley for helpful early conversations and Tony Zador for providing collaborative research environment. Additionally we would like to thank a grant from Schmidt Futures to CSHI. for funding.
Code availability
The code for all figures this paper were written in Jax and are available http://hhwaaooooooooo//////////a
10

---

References
[1] Sanjeev Arora, Elad Hazan, and Satyen Kale. The Multiplicative Weights Update Method: a Meta-Algorithm and Applications. Theory of Compuling, 8(6):121-164, 2012. Publisher:
Theory of Computing.
[2] Dylan R. Ashley, Sina Ghiassian, and Richard S. Sutton. Does the Adam Optimizer Exacerbate Catastrophic Forgetting?, June 2021. arXiv:2102.07686.
[3] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622, PMLR, 2015.
[4] Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. Diversity creation methods: survey and categorisation. Information infiommationfusionn, 6(1):5-20, 2005. Publisher: Elsevier.
[5] Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with cwtrributioo-spcciaiiieei experts. In International Conference on Machine Learning, pages 5383-5395. PMLR, 2023.
[6] Badr-Eddine Cherief-Abdellne
and Pierre Alquier. MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy. In Cheng Zhang, Francisco Ruiz, Thang Bui, Adji Bousso Dieng, and Dawen Liang, editors, Proceedings of The 2nd Symposium on Advances in Approxi- mate Bayesian Inference, volume 1 118 of Proceedings of Machine Learning Research, pages
1-21. PMLR, December 2020.
[7] Lénaic Chizat, Edouard Oyallon, and Francis Bach. On Lazy Training in Differentiable Programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[8] Martin W Cripps. Divisible updating. Manuscript, UCL., 2018.
[9] Francesco D Angelo and Vincent Fortuin. Repulsive Deep Ensembles are Bayesian. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 3451-3465. Curran Associates, Inc., 2021.
[10] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty- guided Continual Learning with Bayesian Neural Networks. In International Conference on Learning Representations, 2020.
[1]] Sebastian Farquhar and Yarin Gal. A unifying bayesian view of continual learning. arXiv preprint arXiv: 0902406494 2019.
[12] Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059,
PMLR, 2016
[13] Ian J. Goodfellow. Mchdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An Empirical Investigation of Catastrophic Forgetting in Stadienn-Baasee Neural Networks, March 2015. arXiv:1312.6211 [cs, stat].
[14] Kazuyuki Hara, Daisuke Saitoh, and Hayaru Shouno. Analysis of dropout learning regarded as ensemble learning. In Artificial Neural Networks and Machine Learning-ICANC 2016: 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part I1 25, pages 72-79. Springer, 2016.
[15] Trevor Hastie, Saharon Rosset, Ji Zhu, and Hui Zou. Multi-class adaboost. Statistics and its Interface, 2(3):349-336)) -600 2009. Publisher: International Press of Boston.
[16] Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian Deep Ensembles via the Neural Tangent Kerne.. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1010-1022. Curran Associates, Inc., 2020.
[17] Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.

---

[18] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts, Neural computation. 3(1):79-88 87, 1991. Publisher: MIT Press.
[19] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel:
Convergence and Generalization in Neural Networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. (Cea-Biaanee and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[20] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181-214 1994, Publisher: MIT Press.
[21] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jachong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D. Yoo. Forget-frre Continual Learning with Winning Subnetworks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10734 10750. PMLR, July 2022.
[22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barbika-Ba Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catas- trophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3222 March 2017. Publisher: Proceedings of the National Academy of Sciences.
[23] Jyrki Kivinen and Manfred K. Warmuth. Exponentiated Gradient versus Gradient Descent for Linear Predictors. Information and Computation, 132(1):1-63, January 1997.
[24] Yajing Kong, Liu Liu, Huanhuan Chen, Janusz Kacprzyk, and Dacheng Tao. Overcoming Catastrophic Forgetting in Continual Learning by Exploring Eigenvalues of Hessian Matrix. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2023. Conference Name: IEEE Transactions on Neural Networks and Learning Systems.
[25] Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P. Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, Anurag Daram, Stefano Fusi, Peter Helfer, Leslie Kay, Nicholas Ketz, 7solt Kira, Soheil Kolouri, Jeffrey I.. Krichmar, Sam Kriegman, Michael Levin, Sandeep Madireddy, Santosh Manicka, Ali Marjaninejad, Bruce McNaughton, Risto Miikkulinenn, Zaneta Navratilova, Tej Pandit, Alice Parker, Praveen K. Pilly, Sebastian Risi, Terrence J. Sejnowski, Andrea Soltoggio, Nicholas Soures, Andreas S. Tolias, Darío Urbina-Meléndez, Francisco J. Valero-Cuevas, Gido M. van de Ven, Joshua T. Vogelstein, Felix Wang. Ron Weiss, Angel Yanguas-Gil, Xinyun Zou, and Hava Siegelmann. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196-210 March 2022. Publisher: Nature Publishing Group.
[26] Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann. Continual Learning with Bayesian Neural Networks for Non- von-Stationary Data. September 2019.
[27] Esther Levin, Naftali Tishby, and Sara A Solla. A statistical approach to learning and generaliza- tion in layered neural networks. Proceedings of the IEFF, (8(10))1568-11 1990. Publisher: IEEE.
(28) Honglin Li, Payam Barnaghi, Shirin Enshaeifar, and Frieder Ganz. Continual learning us- ing bayesian neural networks. IEEE transactions on neural networks and learning systems, 32(9):4243-4252, 2020. Publisher: IEEE.
[29] Chaoyue Liu, Libin Zhu, and Misha Belkin. On the lincarity of large non-linear models: when and why the tangent kernel is constant. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15954-15964 Curran Associates, Inc., 2020.
[30] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. pages 11986-11886, 2022.
[31] Ekdeep Singh Lubana, Puja Trivedi, Danai Koutra, and Robert Dick. How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation. In Sarath Chandar, Razvan Pascanu, and Doina Precup. editors, Proceedings of The 1sl Conference on Lifelong Learning Agents, volume 199 of Proceedings of Machine Learning Research, pages 839-837. PMLR, August 2022.
12

---

[32] David JC MacKay, A practical Bayesian framework for backpropagation networks. Neural computation, 4(3):448 472, 1992. Publisher: MIT Press.
[33] Andres Masegosa. Learning under Model Misspecification: Applications to Variational and Ensemble methods. In Advances in Neural Information Processing Systems, volume 33, pages 5479-5491. Curran Associates, Inc., 2020.
[34] Ron Meir and Tong Zhang. Generalization enror bounds for Bayesian mixture algorithms. Journal of Machine Learning Research, 4(Oct):839-86)) 2003.
[35] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide Neural Networks Forget Less Catastrophically. In Proceedings of the 39th International Conference on Machine Learning, pages 15699-15717. PMLR, June
2022. ISSN: 2640-3498.
[36] Seyed Iann Mirzadeh, Mehrdad Farajtahar, Razvan Pascanu, and Hassan Ghasemzadeh. Un- derstanding the Role of Training Regimes in Continual Learning. In Advances in Neural Information Processing Systems, volume 33, pages 7308-7320. Curran Associates, Inc., 2020.
[37] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational Continual Learning. In International Conference Learning Representations, 2018.
[38) Luis A. Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and Generalization in Neural Network Ensembles. In Proceedings The 25th International Conference on Artificial Intelligence and Statistics, pages 11720-11743, PMLR, May 2022. ISSN: 2640-3498.
[39] Sam Powers, Eliot Xing, and Abhinav Gupta. Self-activating neural ensembles for continual reinforcement learning. In Conference on Lifelong Learning Agents, pages 683-704. PMLR, 2022.
[40] Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022.
[41] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online Structured Laplace Approxi- mations for Overcoming Catastrophic Forgetting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc., 2018
[42] Murray Shanahan, Christos Kaplanis, and Jovana Mitrović. Encoders and ensembles for task-free continual learning, arXiv preprint arXiv:2105. 1att22005.3333.. 13327, 2021.
[43] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gasd gated mixture-of-experts layer. arXiv preprint arXiv: I 000.0..638,... 2017.
[44] Samuel L Smith and Quoc V Le. A Bayesian Perspective on Generalization and Stochastic Gradient Descent. In International Conference on Learning Representations, 2018.
[45] Volker Tresp and Michiaki Taniguchi. Combining Estimators Using Non-Constant Weighting Functions. In Advances Neural Information Processing Systems, volume 7. MIT Press, 1994.
[46] Alexey Tsymbal, Seppo Puuronen, and David W Patterson. Ensemble feature selection with the simple Bayesian classification. Information fusion, 4(2):87-100, 2003. Publisher: Elsevier
[47] N. Ueda and R. Nakano. Generalization error of ensemble estimators. In Proceedings of International Conference on Neural Networks (ICNN"96), volume 1, pages 90-95 vol.1, June 1996.
[48] Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185-11 December 2022. Publisher: Nature Publishing Group.
[49] Joshua T. Vogelstein, Jayanta Dey, Hayden S. Helm, Will LeVine, Ronak D. Mehta, Tyler M. Tomita, Haoyin Xu, Ali Geisa, Qingyang Wang, Gido M. van de Ven, Chenyu Gao, Weiwei Yang, Bryan Tower, Jonathan Larson, Christopher M. White, and Carey E. Priebe. Representation Ensembling for Synergistic Lifelong Learning with Quasilinear Complexity, February 2024.
arXiv:2004.1 12000002119988 [cs, stat].
[50] Max Welling and Yee W 'Thh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pages 681-688, Citeseer, 2011.
13

---

[51] Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swiqtkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How Good is the Bayes Posterior in Deep Neural Networks Really?, July 2020. arXiv:0002.02405 [cs, stat].
[52] Mateusz Andrzej Wójeik, Witold Košciukiewicz, Adam Gonczarek, and Tomasz Jan Kajdanow- icz. Neural Architecture for Online Ensemble Continual Learning. In Sixth Workshop Meta-Learning at the Conference on Neural Information Processing Systems, 2022.
[53] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelligence. Proceedings of machine learning research, 70:3987-3995,, 2017.
[54] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, and others. dixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7111 -11411 2022.
14

---

8 Appendix
8.1 Proof of Theorem 2
Proof. We hegin by noting that the change in weights we can be split up the sign and magnitude,
Aw; Aws|sign(( Aw;).
We will then interpret |wwl as the unnormalized component weight. The remaining terms must be the component functions.
To identify these functions, and show that they satisfy the properties a probability distribution, we will rearrange terms. First, noting that Σ; Aw,l for some constant (potentially 1),
plu,,, w()) = p(ulz, W()) + Σ Awa (sign( (Aw;)
Bw (0) (
+
(17)
Awsl
plyla, w(o)) + sign( Aw:)
Ow oy plyla, w(())
+0(1AW113)
z
p(ala.f))
(18)
N
Awwl
plylor, wil)) (1+ 1 sign(Am;)
i
log jog(yyla (((o)) + ((1AW1)
Ow,
(19)
We call the term p(ylar, xr f&) the neural tangent expert.
The neural tangent expert provides valid probability distribution for small AW. First, see that it satisfies Σ, p(y, (z, ft) 1. This can be seen from the fact that the right term inside p(yl fa) (the parentheses the middle line) sums 0 over the output label:
Op(y, Jx. W())
Σ p(usto 12, fi) = Σ 2 sign( Aw;)
Ow! (0)
= 2 sign Aw;) Σ p(u, w()) d Ooo plu, (r, W())
aw (0)
0
Here we have used the identity that the expectation of a score function is zero
Next, we will show that 2 |æ, (f) 2 O. First, since each p(y, (x, sum to I over j. no component can be greater than if all components are nonnegative. Thus, only needs be shown that p(y)2 2, fi) 2 O. While this cannot be guaranteed in general, by construction we have assumed that 2L 1. This Lipsehitz continuity bounds the L2 norm of the gradients of the log likelihood. which in turn bounds the I.1 norm and implies that any individual gradient has magnitude less than :
a log p(u; Jr. W() 1
Ow: (0)
Thus, whether ign) sign( (w,,))
the term in parenthesis is nonnegative.
log logp(y/z, (0)))) 0.
sign Aw;)
Ow,
(0)
15

---

8.2 Preventing component functions from changing by keeping the network close to initialization
The continual learning ability of Bayesian ensemble derives from learning weight of fixed functions. these functions change over time, then there no guarantee that the likelihood each function at time & still reflects the cumulative likelihood of past data under the current function.
One good way to ensure this does not occur is to ensure that the parameters change as little as possible from initialization. Although it is typical to measure this distance with ||W]],, we instead measure distance as the relative entropy of the change in parameters from the uniform perturbation, due to the simplicity of its result. These have the same minimum; remembering that |&W]| | 1, by normalization, the smallest Euclidean distance ||W]| 2/2 will occur when all Wt are equal.
To derive the maximum maxinum-entropp entropy vector | W| that is as similar as possible to p(fi(D), we will follow the steps of [23]. A first step then find the value that minimizes:
the notion of similarity L between |W] and p(f()D). We will
U((AW]) - ( +
Here 3 is parameter that trades off between entropy and matching p(f[D), and is a Langrange multiplier that ensures the parameters remain normalized.
If one chooses to maximize the dot product AWI' P(fIDD) one obtains the following relation:
e ((11D)
10 e
Alternatieely, if one chooses to minimize the relative entropy (L(IAW).P(fIDD)) then one obtains
D(
E.P(AID
We implement this second term. If the posterior likelihoods are maintained in log space, B acts as multiplicative scale upon the log data likelihood.
8.3 Pseudocode for the NTE update rule using current gradients
Algorithm Neural Tangent Ensemble posterior update rule with current gradients
Receive dataset D {ax, yk Y
Initialize a neural network with parameters W[())
Set learning rate and discount factor 0 < Perturb the network with some AW such that ||W]]] for each example (æt,, yk) 'D do for each edge Wl E W do
Compute the data likelihood for each expert p(yn (Kk fi) sign (Aw,) p(()
aw
Update perturbation multiplicatively Aw; + Aw; p(yn (xx, f:)
end for
Renormalize perturbation such that E, AAw; =
Optionally clip the change in parameters to prevent large changes, such that Awyy end for
8.4 Experiment details
All MNIST experiments were completed on two NVIDIA RTX 6000 cards, and all CIFARIOM experiments were conducted on NVIDIA H100 cards. Over 1,500 individual models were trained across all seeds and conditions, amounting to roughly 440 GPU-hours of compute time.
16

---

8.4.1 Figure 2
A single MLP was trained with 1,000 hidden units per layer and 2 hidden layers using ReLU nonlinearities. The model perturbed from initialization with random normal vector with scale 0.001, and then was trained with the NTE update rule (Algorithm [ but using the Jacobian of the model at initialization. The batch size was and the parameters of the NTE algorithm were v) = 0.01 and == 1.
8.4.2
Figure
We first created a Permuted MNIST task and code to measure the test accuracy on all tasks after training on each task sequentially. All reported results use 5 tasks.
We trained MLP on this task with ReLU nonlinearities and 1,000 hidden units in 2 hidden layers. We used SGD with batch size 128, learning rate 0.01, and momentum swept from 0 to 1. The momentum buffer was not reset between tasks. We report the standard deviation of random seeds.
site 13 in.t Jaan
nex 1p. F-e-))
503
3x10
Adar 000.
pu.p-
201
Theet lose on Tark
efter 1 tedls
2 2x-10"2
6inn loos
lexs
30>
30%
10:1-
10a
20
10
107
10
10a
13a
10
10'
10
10
10
30
107
10
Number parameters
Number of parameterss
Namber parameters
Figure 6: Loss curves for the task in Fig. 4.
8.4.3 Figure
Here, we used the same continual learning task as Figure 3, but swept the width of the two hidden layers from 10 to 10,000, All batch sizes were 128.
17

---

8.5
Additional experiments
90-
800
70-
ScC tacc TAAEN
Fargetting 48%
60-
0001
0.1:
2323
0.1
3x:10
0.21
10
0.00
tesks
0.5
2x
Torgetting
4x100
6x10-2
10-1
Figure 7: For the same task and architecture as the other figures (Permuted MNIST for 5 tasks with ReLUU MLP with two hidden layers and 1, 1,000 hidden units each), we swept the parameters and r in the Algorithm above. The accuracies (top) and losses (bottom) are shown for the first task after 5 total tasks (left), after immediately finishing that task before task switching (middle) and the difference between the two (right). Error bars show the standard deviation across seeds.
No-Me:15
alter Lasks 0100 (10000000 (QULTT)
0.75
0.6
Test acc. tankk
0.70
0.65
0.4
Test
0.50
0.0
0.45
-0.09
0.0
0.2
04
0.6 0.5
1.0
0.0
022
0.4
0.6
0. 02
D.6
0.6
0.5
1.0
Momantum
Morr entum
Morr entum
Figure 8: Effect of momentum in SGD for modern CNN architectures trained on the CIFAR-10 100 task-incremental task. In this task, models trained 10/100 classes at time, and the softmax output layer is masked to only the active classes. Each model is trained for 100 epochs per task, and evaluated on all previous tasks. The two models shown ResNet18 and ConvNeXtTiny. (left) The test set accuracy on the immediately previous task after learning the final task. (middle). The test set accuracy the first task. (right) The difference between the two plots the left.
18

---

Continual learning
Single task learning
1.0
projectlib SimpleMLP
mojjettib.mooebb.iimppee.
0.98
0.8
Test ac.. on task 1 after 1 tasks
Test acc. on task
after 5 tasks
0.97
0.6
0.96
0.95
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Momentum
Momentum
Figure 9: Identically to Fig. we trained a 2L MLP with 1,000 hidden units on the Permuted MNIST task and varied the momentum of SGD. This time, we reset the momemtum buffer between tasks. Interestingly, this introduces a nonmonotonic behavior and one can attain good performance with momentum near 0.99.
Forgatting
0.75
0.60
500 sqp.qtp.tt 3-33
1100 epoccts)
0.70
0.91
00.6-00
0.55
0.50
acc. on previbur
0.20
after Text acc. tack
0.50
0.42
Fongecting
0.0c
0.05
0.40
0.2
0.35
10a
10'
10"
10a
100
107
10a
Number of parameters
Number of parameters
Number parameters
Figure 10: Performance with width for CIFAR-100. We scaled the number of convolutional filters in all layers of a ConvNeXtTiny by constant factor, and then trained the CIFAR-100 task- incremental task for each network. Subpanels represent identical information as Fig. g
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe the contents of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
19

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The main limitations of this work are 1) Theorem is valid only in the limit of small parameter changes, and with the assumption of low network curvature. This assumption stated clearly in the Theorem statement. Then, in the discussion, we mention that weighing experts by their posterior probability is only optimal when the experts are independent, which is not the case. We do not claim this is the optimal weighing strategy.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are listed upfront, and the proofs are complete.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
20

---

4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experiment details are listed in full. Code is also provided that implements all models and optimizers.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code is provided in supplementary information.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new spun-s benchmark).
21

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
htt.ps.cc/pbllc//guudeeeee Ssw.aeeeeeeeooeeeeesseiiassiaase wwwprddeeumiissinnnnii osiliioiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars where present are clearly described as representing the standard deviation over seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
22

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: In the Appendix, we describe the type of GPU used and approximate number of GPU hours used for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////.orrpss.ccc/ppl///eeee
Answer: [Yes]
Justification: All authors have reviewed and agreed to the Code of Ethics, and affirm that this submission adheres.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Although this theoretical work has little direct impact, we mention in the discussion the impacts of continual learning devices upon society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
23

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This model describes no data or models that have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All re-used code and libraries used according their terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects is reported here.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "M3FGjqlipX", "reviewer": "Reviewer_Gy8g", "summary": "The paper suggested that linearized networks (under lazy learning regime) at the initialization can be understood as an ensemble model where each ensemble component is a function parameterized by a single parameter in the network (i.e. if the network has N parameters then the prediction is an ensemble over N functions). The weight of each ensemble is controlled by the posterior weights of each ensemble/expert conditioned on the data in a particular task. Under this regime, when a new task comes in, experts good at previous tasks but bad at this task would be down-weight, as such the ensemble would continuously be adapted to new tasks without losing ability in previous tasks. Interestingly, the paper shows that the posterior update formula is almost identical to SGD update (without momentum). Additionally, the authors proposed a more practical version of the framework under the rich regime, where each ensemble/expert is evolving throughout time, under the assumption that the parameter does not move too far away from the initialization. Lastly, the authors draw three useful insights from the ensemble interpretation, and empirically verifies the insights on 2 layer NN on permuted MNIST.\n\nOverall I find the connection between linearized network, ensemble and continual learning very interesting and novel. The theory is overall sound to me and the empirical results support the theory well.", "questions": "- What does the subscript k (y_k and x_k) mean in the equation at the bottom of page 3.", "limitations": "- It is unclear how the proposed interpretation can be generalized to more complicated networks or settings, e.g. deeper network or when the parameters move far away from the initialization.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The interpretation proposed is novel and interesting.\n\n- The posterior update perspective guarantees that under the proposed NTE theory, the model after seeing multiple tasks sequentially is identical to the one that sees all tasks jointly.\n\n- The problem studied (catastrophic forgetting) is of great importance.\n\n- The theory is mostly sound and the assumptions are reasonable.\n\n- The application of the interpretation gives useful insights, such as momentum causes forgetting.", "weaknesses": "- It is unclear to me, under the Neural Tangent Ensemble theory, how far can the model change from the initialization, if the model is not changing too much, then it is not surprising that the model is not forgetting, since it is not learning a lot information from the training data.\n\n- Similarly, the theory suggest that the model forgets less when getting wider, but the representation learning flexibility would also drop. So the model is learning less while forgetting less. However, good representation learning is crucial for the performance of many deep learning tasks, does it mean there would be inevitable performance v.s. forgetting tradeoff?\n\n- The experiments seem to be of rather small scale: Although sufficient for supporting the theory, it would still be nice to see if the results can be transferred to real world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and recognition of the novelty of our work. \n\nWe would like to clarify some points raised under the weakness section before addressing specific questions. The first two bullet points raised the possibility that, although the infinite-width limit does not forget past tasks, this might not be overly surprising because networks in this limit do not learn much anyways. We partially agree, but in a more limited sense, as we describe below.\n\nBefore responding, however, we want to emphasize that these are related to the general field-wide discussion about the utility of the NTK limit, rather than a comment on our specific contributions in results and insights. Even if this limit is not achieved, our results can provide useful insight. For example, without the proof that a neural network can be seen as an ensemble, one would not be able to describe the conditions under which the classifiers in this ensemble change. Even before the infinite-width limit, there are many strategies for ensuring this to be the case, such as moving in directions of low curvature or using smoother nonlinearities. These possibilities and many others are opened up by our perspective as normative goals for architecture and optimizer design, even if they are not perfectly achieved.\n\n>It is unclear to me, under the Neural Tangent Ensemble theory, how far can the model change from the initialization, if the model is not changing too much, then it is not surprising that the model is not forgetting, since it is not learning a lot information from the training data.\n\nWe would like to slightly push back on the notion that in the NTK regime the network is not learning new information from the data. In the NTK and lazy training literature, it is true that the kernel is fixed in the infinite width limit. Yet there is still a significant learning problem, namely, to decide which kernel basis vectors are useful for a particular task. Similarly, in our ensemble interpretation, the learning problem is to decide which experts are useful. Quite a bit can be learned in this way if there are many experts. (Wider nets provide more functional diversity, mitigating the need to learn novel representations to perform well on a task.) To take a simple example, if the experts included all possible logistic regression classifiers, the ensemble learning problem is equivalent to Bayesian logistic regression. In general, the information learned from the data is the reduction in entropy over the ensemble, which may be significant. Our framework allows us to make this insight clear—there are weightings (i.e. small changes to initialization) that solve Task A well, and there are weightings that solve Task B well, but only the intersection of both weightings will solve both tasks well.\n\n>Similarly, the theory suggest that the model forgets less when getting wider, but the representation learning flexibility would also drop. So the model is learning less while forgetting less. However, good representation learning is crucial for the performance of many deep learning tasks, does it mean there would be inevitable performance v.s. forgetting tradeoff?\n\nWe would like to slightly amend the tradeoff between continual learning and performance to a tradeoff between continual learning and *feature learning*. Yet, as we hold in the previous paragraph, this is not necessarily the same as performance. Our ability to learn without forgetting depends on the representational overlap between sequences of tasks. You are correct that when tasks have little representational overlap, we must trade-off between forgetting and performance. Yet, we stress that increasing the width/capacity of a network ameliorates this trade-off. Wider networks are likely to have more diverse experts, increasing the chance that the appropriate feature representations are available at initialization. More importantly, our theory provides a framework for understanding this trade-off, and our update rule is only one strategy for balancing it with the primary goal of mitigating forgetting. As we highlight in Section 5, the NTE framing suggests more sophisticated strategies for continual learning while limiting changes to the representations/experts. We believe that this work provides an understanding upon which the community can build such methods.\n\n>Q1: What does the subscript k (y_k and x_k) mean in the equation at the bottom of page 3.\n\nA1: The subscript k indexes over individual samples in the training dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAOmvKrZ1q", "reviewer": "Reviewer_MP6D", "summary": "The paper introduces a novel approach to mitigating catastrophic forgetting in neural networks by introducing the concept of Neural Tangent Ensemble (NTE), a formulation interpreting a single neural network as an ensemble of fixed classifiers, leveraging the Neural Tangent Kernel (NTK) framework. This interpretation allows for the derivation of a Bayesian posterior updating rule, which is shown to be equivalent to a scaled and projected form of stochastic gradient descent (SGD). The paper presents theoretical insights into how neural networks can be viewed as Bayesian ensembles, offering an optimization-guided framework for understanding and addressing forgetting in continual learning scenarios.", "questions": "1. Are there any additional experiments that could be conducted to further validate the effectiveness of the NTE approach in more complex or real-world continual learning tasks?\n2. How does the performance of NFE compare with other optimization-based methods?\n3. Could the author elaborate more on Equation(3) and line 78, \"It is easy to create multitask Bayesian ensembles even when tasks are seen sequentially,\" which is not very direct to me?", "limitations": "The authors have outlined the limitations and the potential societal impact of the work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Theoretical rigor: The authors provide a detailed theoretical analysis, including proofs and derivations.\n2. Connection to SGD: The finding that the NTE posterior update rule is connected to SGD is insightful and could have implications for optimization strategies in neural networks\n3. Clear presentation: The paper is clearly presented with fluent logic.", "weaknesses": "To me, the largest limitation of this paper is the empirical validation. Although the paper includes empirical results that support the theoretical claims, the scope of experiments could be expanded. The experiment is implemented on a relatively simple dataset (Permunated-MNIST). Additional experiments on more complex datasets would be appreciated.\n\nBeyond that, the comparison is relatively insufficient. Could the authors provide comparisons with other optimization-based continual learning methods, like OGD or others, to validate the effectiveness of the proposed method? Or are there any other reasons that they can not be directly compared?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the theoretical novelty and generality of the NTE idea. Following these suggestions, we have implemented several more empirical characterizations with modern CNN architectures and the more complicated CIFAR-100 incremental learning task.\n\nWe would like to emphasize that the contributions of our theoretical work can be somewhat separated from the experiments. Our theory provides a rigorous statement about infinite-width networks, which as a novel bridge between ensembles and NNs should be of general interest. Like many theoretical observations, including the NTK paper, this is not directly applicable but instead can be approximated. The approximation we evaluate – which uses the current gradients instead of the gradients at initialization – is designed as an exploration of this limit. Thus, we view this paper more as a new framework for neural network interpretation and a discussion of under what limits networks forget, and less as a paper documenting a new SOTA continual learning algorithm.\n\n>Q1: Are there any additional experiments that could be conducted to further validate the effectiveness of the NTE approach in more complex or real-world continual learning tasks?\n\nA1: Yes, we have expanded our empirical evaluation to include additional continual learning tasks as well as complex, real-world networks. We have now re-evaluated our main results on CIFAR-100 in the task-incremental setting, in which 10 classes are seen at a time, and using modern architectures such as ResNet and ConvNeXt. \n\n>Q2: How does the performance of [NTE] compare with other optimization-based methods?\n\nA2: In Section 3.2, we provide an insightful and surprising result that our NTE-based belief update closely matches SGD updates (with some constraint). So, we should expect that for single task learning, our rule performs just as well as standard SGD training. As such, this will underperform known methods for continual learning for certain networks (e.g. small ones with high curvature), such as EWC. We do not believe that this needs to be shown visually, as it should be clear from the theoretical result.\n\nOur result is instead designed to illustrate how describe how SGD will perform optimally at continual learning in the lazy learning limit. This was not obvious to us before our derivation. As an aside, we wish to emphasize our framework can be used to justify existing SOTA continual learning methods. EWC, for example, moves in directions of low curvature, exactly as needed to keep the tangent experts fixed. \n\n>Q3: Could the author elaborate more on Equation(3) and line 78, \"It is easy to create multitask Bayesian ensembles even when tasks are seen sequentially,\" which is not very direct to me?\n\nA3: This fact is important to understanding the continual learning contribution of our paper. We start from a theoretical framework for describing networks as ensembles of fixed functions, then use Bayesian ensembling to address the continual learning problem. Referring to Fig. 1, given several tasks (e.g. Task A and Task B) and a set of experts, ${f_i}$, different subsets of experts will solve each task well. We seek to find the intersection of these subsets to identify the experts that solve all tasks sufficiently well. As the Lemma in the general comment states, the order in which tasks arrive sequentially is the order of the multiplication of weightings. Consequently, since multiplication is commutative, it does not matter which order we perform the multiplication; we will still arrive at the same final posterior weighting. Given the importance of this argument to our work, we will clarify the logic in the section that you have highlighted by walking through an example with multiple tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sp9QXRjhlh", "reviewer": "Reviewer_q1Zy", "summary": "This is a theoretical research on preventing forgetting by considering a single network trained with a lazy-regime as an ensemble model of multiple functions and adjusting their weights.", "questions": "**(1) Non-linear output**\n\nMany discussions are based on the formulation of Equation (5), and I assume this setting is intended to be used with non-linear transformation such as softmax. However, in my understanding, the NTK does not remain constant in such cases [1]. What are your thoughts on this perspective? If I have misunderstood, please correct me.\n\n**(2) NTK regime**\n\nThere is a lack of discussion regarding the settings of the model used in this study. In order for the NTK to remain constant during training, parameter scaling or output scaling must be set appropriately. In some settings, lazy training would not be realized, even if the width is increased [2]. In the experiments, does the NTK indeed stop changing as the width increases?\n\n**(3) Suboptimality**\n\nAs mentioned in the Discussion section, experts are not actually independent, so the weighting strategy discussed is not actually optimal. In the Discussion section, various ideas are presented for training experts to maintain independence, but these approaches have not been evaluated in this paper. Therefore, it seems important to show that the impact of suboptimality is not significant. Is that possible? If the impact is too large, I feel that the premise of the discussion might be a bit fragile.\n\n\n[1] Liu et al., On the linearity of large non-linear models: when and why the tangent kernel is constant, NeurIPS 2020.\n\n[2] Yang and Hu, Feature Learning in Infinite-Width Neural Networks, ICML2021", "limitations": "Discussion is only effective within a lazy regime. Additionally, experts are required to be independent.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The discussion is based on solid theory. This paper obtained an insight that the posterior update rule for the NTE is equivalent to a scaled and projected form of stochastic gradient descent without momentum. Because of this relationship, this paper successfully demonstrates the disadvantages of using momentum and the advantages of increasing width which may induce lazy training.", "weaknesses": "I feel that the contribution is valuable, but some settings might be difficult to realize in the context of NTK, even with an infinite limit. I might be able to deepen my understanding through the answers to the questions, so please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For Q1 and Q2, there are really two possible statements in question. The first is whether a particular network at any single moment in time can be seen as an ensemble of valid classifiers. Theorem 1 describes how this is always the case as long as the network is Lipschitz, as one could construct a seed point for the Taylor expansion arbitrarily close to the network. This result stands alone, and could be used in further work to design quantifications of uncertainty, etc. \n\nThe second question is whether and when these classifiers evolve in time due to optimization. We agree that we did not adequately expand on this question. In revision, we will include a discussion of this issue and include our responses to these questions.\n\n> Q1: Many discussions are based on the formulation of Equation (5), and I assume this setting is intended to be used with non-linear transformation such as softmax. However, in my understanding, the NTK does not remain constant in such cases [1]. \n\nYes, in our setting we deal with softmax nonlinearities. Thank you for this reference. We agree that the presence of this output nonlinearity adds structure to the Hessian which does not diminish with width resulting in changes to the NTK kernel for standard initializations. However, we want to emphasize that this does not affect Theorem 1 in itself, which again, relates to whether a linearized network with an arbitrary ∆W can be interpreted as an ensemble. This only relies on the Lipschitz continuity of the network, and one can always choose a perturbation ∆W arbitrarily close to the initial point such that the linearization remains valid. Thus, this question is about how much the tangent experts change - not whether they are indeed interpretable as experts.\n\nIt is a second matter whether the ∆W given by a specified learning algorithm, data, and objective will keep the Jacobian fixed. The results from [1] do indeed suggest that nonlinearities will make this more difficult and perhaps not be achievable even in the infinite width limit. (As an aside, even for softmax nonlinearities a truly lazy regime can still be achieved by scaling the outputs as in Chizet et al, which the authors of [1] mention in their Appendix A.) Even when experts change, minimizing this change remains a goal to reduce forgetting. This can be achieved by making networks wider, but also through other strategies such as the nonlinearities or moving preferentially in directions of low curvature. \n\nIn response to Q1 and Q2, we empirically evaluated whether width scales how much experts change under the NTE update rule. We trained several MLPs of increasing width on MNIST. For each expert, we then calculated how much it changed from initialization via the squared difference of its respective column in the Jacobian from initialization. (Each entry in an expert’s column relates to its effect upon an output probability). We then reported the average of this distance over all experts, and this plot is attached in the pdf. We found that increasing width leads to a diminishing change in the experts, meaning that wider networks will indeed forget less.\n\n> Q2: There is a lack of discussion regarding the settings of the model used in this study. In order for the NTK to remain constant during training, parameter scaling or output scaling must be set appropriately. In some settings, lazy training would not be realized, even if the width is increased [2]. In the experiments, does the NTK indeed stop changing as the width increases?\n\nWe will add a discussion of this important issue. This relates to our paper's central message: keeping the NTK fixed is crucial for continual learning. Thus, in practice, every effort should be made to ensure the network learns lazily. In principle, this could even be used as the basis for the design of new initializations that forget less. \n\nA related possibility is that one might try to derive a rule to weigh the ensemble while remaining in the NTK regime. This motivates the use of the regularizer $\\eta$, which is currently derived in Appendix 8.2. In Section 4.1 and 4.2, we then motivate a rule where the Jacobian around the current weights is used for linearization. This matches the reasoning in [2] that a network must not only weigh the experts (features) at initialization, but also learn useful experts (features) for the task at hand. Furthermore, in Section 5.4, we explain that moving along low curvature directions allows the network to stay in a linear regime. This aligns with the results of [1] where a transition to NTK constancy is given by a Hessian with shrinking spectral norm. While we cannot guarantee that all directions of the loss landscape become shallow with increasing width, we can choose only to move in the directions that do, effectively matching the settings of [1] in practice. Implementing a practical rule under this guidance is out of the scope of our paper and left for future work.\n\n>Q3: As mentioned in the Discussion section, experts are not actually independent, so the weighting strategy discussed is not actually optimal...\n\nA3: The recognition that this weighing strategy is suboptimal was an interesting finding to us – especially given our finding that the posterior update rule is remarkably similar to SGD. Thus, due to our derivation, we are able to see that this SGD-like rule is in fact suboptimal. We think it would be an interesting line of future work to actually improve upon SGD by taking the ensemble interpretation of wide NNs and penalizing experts for their redundancy.\n\nFor the time being, however, our empirical results show that this suboptimality does not prohibit taking this overall approach. We are able to learn tasks with matched performance with standard optimization while maintaining the ability to continually learn. Furthermore, we would like to highlight again that the posterior update rule for the NTE is, surprisingly (to us), very similar to SGD, which in practice of course works quite well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MxoZbaBLjO", "reviewer": "Reviewer_Yq9r", "summary": "This is a theoretical paper that seeks to find an algorithm to train neural networks without forgetting in continual learning settings.\n\nThe authors show that in the infinite width limit, a single classification network can be reformulated as a weighted ensemble of fixed classifiers (fixed experts).\n\nThey provide a learning rule, which computes the posterior distribution of each expert.\n\nThis learning rule turns out to be similar to SGD, where there is a per-parameter learning rate and the different of the parameters to their initial state is constrained to be L1 normalized (unsure of the exact correct phrasing of the L1 projection step).\n\nIn the finite case, the experts can no longer be viewed as static. \n\nExperiments show that any amount of momentum hurts continual learning performance, but momentum does help when learning a single task from scratch.", "questions": "What happens if there is label noise? This analysis seems to assume that the dataset is perfect, which raises the question: if you have an idealized network that cannot forget, what is the impact of a mislabeled example?\n\nThe paper claims that in the infinite-width network each parameter/weight/edge contributes a classifier to the ensemble. This seems to imply that weights in earlier layers are included here, is that true? I'm confused about how to picture this. In the first layer, there is a weight connecting one of the finite input neurons to one of the infinite hidden neurons. That particular edge isn't able to see much information, but its output does connect to an infinite number of neurons in the second hidden layer. Is its interpretation as a weight on the neural-tangent-experts somehow implicit wrt to the rest of the network depth? Is it that the edge allows energy to accumulate in the \"right spots\"?\n\nSimilarly, in the context of finite networks, is there a way to extract the individual experts? Are the  number of experts still equal to the number of model parameters in this case?\n\n\nFigure 3: I'm a confused about the continual learning task setup for permuted MNIST. It would be useful to remind readers what the mechanism for predicting on task 1 after 5 tasks is. Is there a new head? Do the number of output units stay fixed throughout all tasks?\n\nIn theorem 1: why is the perturbation important?  Is ΔW the linear update that brings an untrained NTK weights (W⁰) to the trained NKT weights (Wᵗ)? If this is true, it might help to the statement of theorem 1 to call that out.\n\nCan anything be said theoretically about how fast increasing the width of a network converges towards the NTE limit?\n\nIn Figure 4 are the curves over just the first task? In other words, is this just training on regular MNIST to show how the NTE update rule begins to fail as the weights drift from their initial state?", "limitations": "The authors are clear about the technical limitations of the work, and do not\nmake any unreasonable claims.\n\nCode to reproduce looks like it exists.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Theoretical understanding of how to prevent forgetting in networks as training continues is important and helps move the field forward.\n\nThe connection of SGD to posterior updates is interesting and (as far as I'm aware) novel.\n\nThe experiments exploring the effect of momentum on a networks ability to continually learn are interesting and useful from a practitioner's perspective.\n\nExperiments demonstrate that the NTE update rule becomes more appropriate as width increases.\n\nThe work opens up future directions that may be able to estimate uncertainty based on the variance of experts under the NTE formulation.", "weaknesses": "The claim that Bayesian ensembles of fixed functions are natural continual learners does not seem obvious to me. In the infinite-width I buy it, but for finite networks, it seems like it would be possible to prune all of the experts away leaving you with a degenerate model.\n\nOnly one continual learning benchmark dataset (permuted MNIST) is used in the experiments. The details of this task are not well explained.\n\nThe introduction highlights the value of Bayesian posterior updates as being order-independent. However, there is no experiment exploring the sensitivity of the NTE update rule to the ordering of the data. \n\nSection 5.4 seems out of place. It isn't a result, and it seems a stretch to call it a prediction. It may be useful reducing it to a few sentences in the related work section so other sections can include more clarifying text.\n\nOverall, the proofs seem correct, but that required a significant amount of time and further reading for me to come to that conclusion. Such theoretical work would be much stronger if it codified its proofs in the form of a proof assistant like Lean4 or Coq. This would require the reviewer to only check that the claim is correctly formulated. The proof itself would be verified automatically, and there would be no question about its correctness.  Codified proofs would allow readers learn more about any unfamiliar identities used with full confidence and remove all ambiguity stemming from unfamiliar notation.\n\nTypo: Line 205. \"This is constrains\" should be \"This constrains\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The claim that Bayesian ensembles of fixed functions are natural continual learners does not seem obvious to me....\n\nWe have added a Lemma that describes more precisely why weighing fixed functions according to their posterior probability is a learning strategy which does not depend on the order of the data.\n\nTo your second point, since the weights reflect the posterior probability, they must sum to 1. This makes it impossible to prune away all of the experts. However, there does exist a degenerate case where only 1 expert remains, yet even in this case the weight $w=1$ on this model is invariant to data ordering\n\n>Only one continual learning benchmark dataset (permuted MNIST) is used in the experiments...\n\nWe have added an additional benchmark of CIFAR-100 in the task-incremental setting in which only 10 output classes are observed at a given time. We will add further experimental details to the revised manuscript.\n\n>The introduction highlights the value of Bayesian posterior updates as being order-independent...\n\nWe hope that the Lemma above is sufficient to clarify this point. This is a guarantee of the fact that the Bayesian posterior is obtained via the likelihood of the data under each expert, which depends on data likelihood multiplicatively.\n\n>Section 5.4 seems out of place...\n\nWe agree and will move this text.\n\n>Overall, the proofs seem correct.... Such theoretical work would be much stronger if it codified its proofs in the form of a proof assistant like Lean4 or Coq...\n\nThank you for this detailed reading! For the current manuscript, we regret that we did not have sufficient time in the rebuttal period to codify this work in a proof assistant.\n\n>Q1: What happens if there is label noise? ...\n\nA1: This is an important point. This is a drawback of this problem setting: the goal of being invariant to the order of data inherently treats all datapoints equally. We note that this is a general feature of this domain, also a problem of full-dataset Gradient Descent, and that it is standard to train networks by the likelihood of the dataset under the model. However, it is an interesting line of research to combine robustified learning algorithms with continual learning.\n\n>Q2: ... This seems to imply that weights in earlier layers are included here, is that true? ... \n\nA2: Indeed, each edge contributes an expert – even the lower layers. This is true both for finite networks and infinite-width networks. This is indeed surprising – as at first glance it seems odd that input layers should be contributing a probability distribution over the output classes. \n\nWe find it helpful to imagine gradient flow. Take some edge E in the middle of the network. There are many input/output paths that traverse edge E as they ascend from the inputs to the outputs. The weight of edge E acts as a gain control on all of these input/output paths. When we slightly perturb edge E, the effect upon the output classes due the sum of all of these paths slightly changes as well. This is how the architecture of the remaining network enters: the effect of edge E depends on gradient flow through all upstream and downstream edges. It is this perturbative effect upon the output classes that gives edge E its additive effect upon the output logits. Theorem 1 then states further that these perturbations themselves act as a valid classifier, thus allowing us to interpret the entire network as an ensemble over the perturbative effects of each edge in the entire network.  \n\nWe can state this slightly more formally. The effect of a single edge (expert) on the output of the network is given w.r.t. the other edges across all layers in the network. Specifically, in the NTK, the Jacobian controls how each edge affects the output. Consider a single entry of the Jacobian, $df_i / dw_j$, which controls how the perturbation around edge $w_j$ adjusts a single output of the network. This quantity is the gradient path from $f_i$ to $w_j$. For finite networks, a single column of the Jacobian can be understood as an expert, and the number of experts (length of the column) is equal to the number of parameters.\n\n>Q3: Figure 3: I'm a confused about the continual learning task setup for permuted MNIST....\n\nA3: For permuted MNIST, nothing changes about the network from task to task since the output labels are always the same. Only the pixels in the input are permuted. \n\nAdditionally, we will clarify our continual learning setups by using the vocabulary of van de Ven et al. 2022 (https://www.nature.com/articles/s42256-022-00568-3). The Permuted MNIST task is an example of domain-incremental learning. In contrast, the CIFAR-100 task which shows 10 classes at a time is an example of task-incremental learning. \n\n>Q4: In theorem 1: why is the perturbation important? ...\n\nA4: Yes, $ΔW$ is the change from the untrained weights to the trained weights. It is important for Theorem 1, because the magnitude of the perturbation is the quantity that we define as the posterior probabilities (i.e. weights on the experts). \n\nIn general, the perturbation is key due to our reliance on a first-order Taylor expansion describing the network at $f(W+∆W)$. Coarsely, in order for a network with N edges to be a Mixture of Experts with N experts, there needs to appear a $\\sum_i^N$.\n\n>Q5: Can anything be said theoretically about how fast increasing the width of a network converges towards the NTE limit?\n\nA5: This is a complex question. We are aware of some work describing how quickly networks approach the lazy regime, but only in certain settings. We have attached an empirical study of how quickly the NTE limit is achieved with width for MLPs on the MNIST task. \n\n>Q6: In Figure 4 are the curves over just the first task? ...\n\nA6: Might you mean Figure 2? For Figure 2, this a network just trained on regular MNIST to show the effect that you describe, and yes, this is just the first task it has seen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We were happy to see that all 4 reviewers found our primary contribution – that linearized classifier networks are ensembles of experts (Theorem 1) – to be interesting, novel, well-supported, and relevant to problems of importance. Likewise, the reviewers appreciated the surprising nature and importance of our second theoretical result, which states that the posterior belief update rule for this ensemble is similar to SGD. The reviewers then noted several areas for improvement, and we believe the changes we have made in response have greatly strengthened the paper.

One common request was to clarify why an invariance to the ordering of data (thus solving continual learning) emerges when learning with an ensemble of fixed, independent experts. To address this, we have added a Lemma proving this fact. This can be found at the end of this general response.

Our empirical evaluations demonstrate that the ensemble provides useful insights that justify both known and novel observations about forgetting in NNs. However, all reviewers rightly pointed out that these evaluations were limited, leaving it unclear whether they would generalize to more complex networks and tasks. We have addressed this concern by adding additional continual learning tasks, datasets, and architectures. In particular, we evaluate a setup based on CIFAR-100 where each task is to predict a distinct subset of classes in the full dataset. We repeat our studies on modern convolutional networks such as ConvNeXt. Finally, we add additional experiments to address specific questions posed by reviewers, which we discuss in more detail in our individual responses.

Finally, we wish to further emphasize that our theorems provide a general result that will be useful for many subfields beyond continual learning. This work represents a new portal between two well-established domains. It enables a long history of theoretical results and strategies about mixtures of experts and ensembles to be applied to understand and improve neural networks. For example, this work provides a new path for uncertainty estimation, understanding bias/variance tradeoffs and generalization, sparsifying inference, and for architecture and initialization design—none of which we could fully investigate here. 

Moreover, this result provides an intuitive starting point to reason about the transition from rich to lazy regimes. Rather than require the machinery of kernel regression, you can instead see neural networks instead as mixtures of experts. The rich regime corresponds to improving each expert, whereas the lazy regime corresponds to just weighing each expert but keeping them fixed. This understandable analogy can be taken in many directions, both for deep learning in theory and practice.

We believe these results are quite relevant for continual learning, which is why we framed this paper this way. Yet we are excited at the numerous open paths, both within continual learning and in other domains. 

—---------------------------------------------------

Here we formalize the general statement that when a model class is a weighted ensemble of fixed, independent probabilistic classifiers, there is no catastrophic forgetting problem. This key fact motivates our assessment of under what conditions neural networks approach this setting.

When we update the weights of experts based on new data by Bayesian posterior updating, the end result is invariant to the ordering of data due to the multiplicative nature of probability updating. This is restated in the following Lemma, which we will include in the manuscript.

**Lemma: Invariance to data ordering in Bayesian Ensembles.**
Let $\mathcal{F} = {f_1, ..., f_N}$ be a set of fixed experts, $\mathcal{W}=w_1,...,w_N$ be their weights, and $\mathcal{D} = {D_1, ..., D_T}$ be a sequence of datasets from $T$ tasks. Let $w_i=p(f_i|\mathcal{D})$ be the posterior probability of expert $f_i$ given data $\mathcal{D}$.
Then, for any permutation $\pi$ of the indices {1, ..., T},
$p(f_i|\mathcal{D}) = p(f_i|D_1, ..., D_T) = p(f_i|D_{\pi(1)}, ..., D_{\pi(T)})$

*Proof*.
By Bayes' rule, $p(f_i|\mathcal{D}) \propto p(f_i) \prod_{t=1}^T p(D_t|f_i)$. The right-hand side is a product of terms, one for each dataset. Since multiplication is commutative, $\prod_{t=1}^T p(D_t|f_i) = \prod_{t=1}^T p(D_{\pi(t)}|f_i)$ for any permutation $\pi$. Therefore, $p(f_i|D_1, ..., D_T) = p(f_i|D_{\pi(1)}, ..., D_{\pi(T)})$.

Meta-review: The paper has several interesting theoretical contributions, mainly new insights on interpreting a model as ensemble of classifiers and analyzing forgetting. The experimental design is interesting and further supports the theoretical findings. I recommend accept.
Decision: Accept (spotlight)