Year: 2021

Abstract:
Sequential training from task to task is becoming one of the major objects in deep learning applications such as continual learning and transfer learning. Nevertheless, it remains unclear under what conditions the trained model’s performance improves or deteriorates. To deepen our understanding of sequential training, this study provides a theoretical analysis of generalization performance in a solvable case of continual learning. We consider neural networks in the neural tangent kernel (NTK) regime that continually learn target functions from task to task, and investigate the generalization by using an established statistical mechanical analysis of kernel ridge-less regression. We first show characteristic transitions from positive to negative transfer. More similar targets above a specific critical value can achieve positive knowledge transfer for the subsequent task while catastrophic forgetting occurs even with very similar targets. We also reveal that even for continual learning of the same target function, the trained model shows some transfer and forgetting depending on the sample size of each task. We can guarantee that the generalization error monotonically decreases from task to task for equal sample sizes while unbalanced sample sizes deteriorate the generalization. We respectively refer to these improvement and deterioration as self-knowledge transfer and forgetting, and empirically confirm them in realistic training of deep neural networks.

heading: 1 INTRODUCTION
text: As deep learning develops for a single task, it enables us to work on more complicated learning frameworks where the model is sequentially trained on multiple tasks, e.g., transfer learning, curriculum learning, and continual learning. Continual learning deals with the situation in which the learning machine cannot access previous data due to memory constraints or privacy reasons. It has attracted much attention due to the demand on applications, and fundamental understanding and algorithms are being explored (Hadsell et al., 2020). One well-known phenomenon is catastrophic forgetting; when a network is trained between different tasks, naive training cannot maintain performance on the previous task (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017).
It remains unclear in most cases under what conditions a trained model’s performance improves or deteriorates in continual learning. Understanding its generalization performance is still limited (Pentina & Lampert, 2014; Bennani et al., 2020; Lee et al., 2021). For single-task training, however, many empirical and theoretical studies have succeeded in characterizing generalization performance in over-parameterized neural networks and given quantitative evaluation on sample-size dependencies, e.g., double descent (Nakkiran et al., 2020). For further development, it will be helpful to extend the analyses on single-task training to sequential training on multiple tasks and give theoretical backing.
To deepen our understanding of continual learning, this study shows a theoretical analysis of its generalization error in the neural tangent kernel (NTK) regime. In more detail, by extending a statistical mechanical analysis of kernel ridge-less regression on single-task training (Bordelon et al., 2020) to continual learning, we investigate the learning curve, i.e., the dependence of generalization error on sample size or number of tasks. We focus on the continual learning with explicit task boundaries. Suppose that the model continually learns data generated by two similar teacher functions, which we call target functions, from one task to another. All input samples are generated from the
same distribution in an i.i.d. manner, and each task has output samples (labels) generated by its own target function. In the NTK regime, the training of the network becomes independent among heads, and learning is formulated as a regression of each head on a corresponding target function. Our main contributions are summarized as follows. First, we revealed characteristic transitions from negative to positive transfer depending on the target similarity. More similar targets above a specific critical value can achieve positive knowledge transfer (i.e., better prediction on the subsequent task than training without the first task). Compared to this, backward transfer (i.e., prediction on the previous task) has a large critical value, and subtle dissimilarity between targets causes negative transfer. The error can rapidly increase, which clarifies that catastrophic forgetting is literally catastrophic (Section 4.1).
Second, we considered continual learning of the same target function. Even for the continual learning in iterative tasks, the trained model’s performance improves or deteriorates in a non-monotonic way. This depends on the sample size of each task; for equal sample sizes, we can guarantee that the generalization error monotonically decreases from task to task (Section 4.2 for two tasks & Section 5 for more tasks). Unbalanced sample sizes, however, deteriorates generalization (Section 4.3). We refer to these improvement and deterioration of generalization as self-knowledge transfer and forgetting, respectively. Finally, we empirically confirmed that self-knowledge transfer and forgetting actually appear in continual learning of multi-layer perceptron (MLP) and ResNet-18 (Section 5.1). Thus, this study sheds light on fundamental understanding and the universal behavior of continual learning in over-parameterized learning machines.

heading: 2 RELATED WORK
text: Method of analysis. As an analysis tool, we use the replica method originally developed for statistical mechanics. Statistical mechanical analysis enables us typical-case evaluation, that is, the average evaluation over data samples or parameter configurations. It sometimes provides us with novel insight into what the worst-case evaluation has not captured (Abbaras et al., 2020; Spigler et al., 2020). The replica method for kernel methods has been developed in Dietrich et al. (1999), and recently in Bordelon et al. (2020); Canatar et al. (2021). These recent works showed excellent agreement between theory and experiments on NTK regression, which enables us to quantitatively understand sample-size dependencies including implicit spectral bias, double descent, and multiple descent.
Continual learning. Continual learning dynamics in the NTK regime that we investigate was first formulated by Bennani et al. (2020); Doan et al. (2021), though the evaluation of generalization remains unclear. They derived an upper bound of generalization via the Rademacher complexity, but it includes naive summation over single tasks and seems conservative. In contrast, our typical-case evaluation enables us to find such rich behaviors as negative/positive transfer and self-knowledge transfer/forgetting we newly find. The continual learning in the NTK regime belongs to so-called single-head setting (Farquhar & Gal, 2018), and it allows the model to revisit the previous classes (target functions) in subsequent tasks. This is complementary to earlier studies on incremental learning of new classes and its catastrophic forgetting (Ramasesh et al., 2020; Lee et al., 2021), where each task includes different classes and does not allow the revisit. Note that the basic concept of continual learning is not limited to incremental learning but allows the revisit (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017). Under the limited data acquisition or resources of memory, we often need to train the same model with the same targets (but different samples) from task to task. Therefore, the setting allowing the revisit seems reasonable.

heading: 3 PRELIMINARIES

heading: 3.1 NEURAL TANGENT KERNEL REGIME
text: We summarize conventional settings of the NTK regime (Jacot et al., 2018; Lee et al., 2019). Let us consider a fully connected neural network f = uL given by
ul = σwWlhl−1/ √ Ml−1 + σbbl, hl = φ(ul) (l = 1, ..., L), (1)
where we define weight matrices Wl ∈ RMl×Ml−1 , bias terms bl ∈ RMl , and their variances σ2w and σ2b . We set random Gaussian initialization Wl,ij , bl,i ∼ N (0, 1) and focus on the mean squared error loss: L(θ) = ∑N µ=1 ‖yµ − f(xµ)‖2, where the training samples {xµ, yµ}Nµ=1 are composed of
inputs xµ ∈ RD normalized by ‖xµ‖2 = 1 and labels yµ ∈ RC . The set of all parameters is denoted as θ, and the number of training samples is N . Assume the infinite-width limit for hidden layers (Ml →∞), finite sample size, and depth compared with the width. The gradient descent dynamics with a certain learning rate then converges to a global minimum sufficiently close to the random initialization. This is known as the NTK regime, and the trained model is explicitly obtained as
f (c)(x′) = f (c) 0 (x ′) + Θ(x′, X)Θ(X)−1(y(c) − f (c)0 (X)) (c = 1, ..., C). (2) We denote the NTK matrix as Θ, arbitrary input samples (including test samples) as x′, and the set of training samples as X . The indices of f mean 0 for the model at initialization and c for the head of the network. Entries of NTK Θ(x′, x) are defined by∇θf0(x′)∇θf0(x)>. We write Θ(X) as an abbreviation for Θ(X,X). The trained network is equivalent to a linearized model around random initialization (Lee et al., 2019), that is, f (c) = f (c)0 +∇θf (c) 0 ∆θ with
∆θ = θ − θ0 = C∑ c=1 ∇θf (c)0 (X)>Θ(X)−1(y(c) − f (c) 0 (X)). (3)
While over-parameterized models have many global minima, the NTK regime’s dynamics implicitly select the above θ, which corresponds to the L2 min-norm solution. Usually, we ignore f0 by taking the average over random initialization. The trained model (2) is then equivalent to the kernel ridge-less regression (KRR).
NTK regime also holds in various architectures including ResNets and CNNs (Yang & Littwin, 2021), and the difference only appears in the NTK matrix. Although we focus on the fully connected network in synthetic experiments, the following NTK formulation of continual learning and our theoretical results hold in any architecture under the NTK regime.
Continual learning. We denote the set of training samples in the n-th task as (Xn, yn) (n = 1, 2, ...,K), a model trained in a sequential manner from task 1 to task n as fn and its parameters as θn. That is, we train the network initialized at θn−1 for the n-th task and obtain fn. Assume the number of tasks K is finite. The trained model within the NTK regime is then given as follows (Bennani et al., 2020; Doan et al., 2021):
fn(x ′) = fn−1(x ′) + Θ(x′, Xn)Θ(Xn) −1(yn − fn−1(Xn)), (4)
θn − θn−1 = ∇θf0(Xn)>Θ(Xn)−1(yn − fn−1(Xn)). (5) We omit the index c because each head is updated independently. The model fn completely fits the n-th task, i.e., yn = fn(Xn). The main purpose of this study is to analyze the generalization performance of the sequentially trained model (4). At each task, the model has an inductive bias of KRR in the function space and L2 min-norm solution in the parameter space. The next task uses this inductive bias as the initialization of training. The problem is whether this inductive bias helps improve the generalization in the subsequent tasks.
Remark (independence between different heads). For a more accurate understanding of the continual learning in the NTK regime, it may be helpful to remark on the heads’ independence, which previous studies did not explicitly mention. As in Eq. (2), all heads share the same NTK, and f (c) depends only on the label of its class y(c). While the parameter update (3) includes information of all classes, the c-th head can access only the information of the c-th class1. For example, suppose that the n-th task includes all classes except 1, i.e., {2, ..., C}. Then, the model update on the the class 1 at the n-th task, i.e., f1n − f1n−1, becomes
∇θf (1)0 (x′)(θn − θn−1) = ∇θf (1) 0 (x ′) ∑ c=2 ∇θf (c)0 (Xn)>Θ(Xn)−1(y(c)n − f (c) n−1) = 0
because ∇θf (c)0 ∇θf (c′)> 0 = 0 (c 6= c′) in the infinite-width limit (Jacot et al., 2018; Yang, 2019). Thus, we can deal with each head independently and analyze the generalization by setting C = 1 without loss of generality. This indicates that in the NTK regime, interaction between different heads do not cause knowledge transfer and forgetting. One may wonder if there are any non-trivial knowledge transfer and forgetting in such a regime. Contrary to such intuition, we reveal that when the subsequent task revisits previous classes (targets), the generalization shows interesting increase and decrease.
1We use the term ”class”, although the regression problem is assumed in NTK theory. Usually, NTK studies solve the classification problem by regression with a target y(c) = {0, 1}.

heading: 3.2 LEARNING CURVE ON SINGLE-TASK TRAINING
text: To evaluate the generalization performance of (4), we extend the following theory to continual learning. Bordelon et al. (2020) obtained an analytical expression of the generalization for NTK regression on a single task (2) as follows. Assume that training samples are generated in an i.i.d. manner ( xµ ∼ p(x)) and that labels are generated using a square integrable target function f̄ :
f̄(x) = ∞∑ i=0 w̄iψi(x), y µ = f̄(xµ) + εµ (µ = 1, ..., N), (6)
where w̄i are constant coefficients and ε represents Gaussian noise with 〈εµεν〉 = δµνσ2. We define ψi(x) :=
√ ηiφi(x) with basis functions φi(x) given by Mercer’s decomposition:∫
dx′p (x′) Θ (x, x′)φi (x ′) = ηiφi(x) (i = 0, 1, . . . ,∞). (7)
Here, ηi denotes NTK’s eigenvalue and we assume the finite trance of NTK ∑ i ηi < ∞. We set η0 = 0 in the main text to avoid complicated notations. We can numerically compute eigenvalues by using the Gauss-Gegenbauer quadrature. Generalization error is expressed by E1 := 〈∫ dxp(x) ( f̄(x)− f∗(x) )2〉 D
where f∗ is a trained model and 〈· · · 〉D is the average over training samples. Bordelon et al. (2020) derived a typical-case evaluation of the generalization error by using the replica method: for a sufficiently large N , we have asymptotically
E1 = 1
1− γ ∞∑ i=0 ηiw̄ 2 i ( κ κ+Nηi )2 + γ 1− γ σ2. (8)
Although the replica method takes a large sample size N , Bordelon et al. (2020); Canatar et al. (2021) reported that the analytical expression (8) coincides well with empirical results even for small N . The constants κ and γ are defined as follows and characterize the increase and decrease of E1:
1 = ∞∑ i=0 ηi κ+Nηi , γ = ∞∑ i=0 Nη2i (κ+Nηi)2 . (9)
The κ is a positive solution of the first equation and obtained by numerical computation. The γ satisfies 0 < γ < 1 by definition. For N = αDl (α > 0, l ∈ N, D 1), we can analytically solve it and obtain more detailed evaluation. For example, a positive κ decreases to zero as the sample size increases and E1 also decreases for σ2 = 0. For σ2 > 0, the generalization error shows multiple descent depending on the decay of eigenvalue spectra. Since multiple descent is not a main topic of this paper, we briefly summarize it in Section A.5 of the Supplementary Materials.

heading: 4 LEARNING CURVES BETWEEN TWO TASKS
text: In this section, we analyze continual learning in the NTK regime (4) between two tasks (K = 2). One can also regard this setting as transfer learning. We sequentially train the model from task A to task B, and each one has a target function defined by
f̄A(x) = ∑ i w̄A,iψi(x), f̄B(x) = ∑ i w̄B,iψi(x), [w̄A,i, w̄B,i] ∼ N (0, ηi [ 1 ρ ρ 1 ] ). (10)
The target functions are dependent on each other and belong to the reproducing kernel Hilbert space (RKHS). By denoting the RKHS by H, one can interpret the target similarity ρ as the inner product 〈f̄A, f̄B〉H/(‖f̄A‖H‖f̄B‖H) = ρ. These targets have dual representation such as f̄(x) =∑ i αiΘ(x ′ i, x) with i.i.d. Gaussian variables αi (Bordelon et al., 2020), as summarized in Section E.1. We generate training samples by yµA = f̄A(x µ A) + ε µ A (µ = 1, ..., NA) and y µ B = f̄B(x µ B) + εµB (µ = 1, ..., NB), although we focus on the noise-less case (σ = 0) in this section. Input samples xµA and x µ B are i.i.d. and generated by the same distribution p(x). We can measure the generalization error in two ways: generalization error on subsequent task B and that on previous task A:
EA→B(ρ) = 〈∫ dxp(x)(f̄B(x)− fA→B(x))2 〉 , (11)
EbackA→B(ρ) = 〈∫ dxp(x)(f̄A(x)− fA→B(x))2 〉 , (12)
where we write f2 as fA→B to emphasize the sequential training from A to B. The notation EbackA→B(ρ) is referred to as backward transfer (Lopez-Paz & Ranzato, 2017). Large negative backward transfer is known as catastrophic forgetting. We take the average 〈· · · 〉 over training samples of two tasks {DA,DB}, and target coefficients w̄. In fact, we can set w̄ as constants, and it is unnecessary to take the average. To avoid complicated notation, we take the average in the main text. We then obtain the following result. Theorem 1. Using the replica method under sufficiently large NA and NB , for σ = 0, we have
EA→B(ρ) = ∑ i
[ 2(1− ρ) (1− qA,i) +
q2A,i 1− γA
] EB,i, (13)
EbackA→B(ρ) = ∑ i [ 2(1− ρ)(1 + FγB (qA,i, qB,i))η2i + q2A,i 1− γA EB,i ] , (14)
where we define qA,i = κA/(κA +NAηi), qB,i = κB/(κB +NBηi), EB,i = q2B,iη 2 i /(1− γB) and FγB (a, b) = b(a− 2) + b2(1− a)/(1− γB).
The constants κA and γA (κB and γB , respectively) are given by setting N = NA(NB) in (9). The detailed derivation is given in Section A. Technically speaking, we use the following lemma: Lemma 2. Denote the trained model (2) on single task A as fA = ∑ i w ∗ A,iψi, and define the
following cost function: E = 〈∑
i φi(w ∗ A,i − ui)2 〉 DA
for arbitrary constants φi and ui. Using the replica method under a sufficiently large NA, we have
E = ∑ i [ (w̄A,i − ui)2 − 2w̄A,i(w̄A,i − ui)qA,i + { w̄2A,i + ηiNA κ2A (EA + σ 2) } q2A,i ] φi,
where EA denotes generalization error E1 on single task A.
For example, we can see that EA is a special case of E with φi = ηi and u = w̄A, and that EA→B(ρ) is reduced to u = w̄B and φi = q2B,iη 2 i /(1− γB) after certain calculation.
Spectral Bias. The generalization errors obtained in Theorem 1 are given by the summation over spectral modes like the single-task case. The EB,i corresponds to the i-th mode of generalization error (8) on single task B. The study on the single task (Bordelon et al., 2020) revealed that as the sample size increases, the modes of large eigenvalues decreases first. This is known as spectral bias and clarifies the inductive bias of the NTK regression. In more detail, they showed that when D is sufficiently large and NB = αDl (α > 0, l ∈ N), the error of each mode is asymptotically given by EB,i = 0 (i < l) and η2i (i > l). We see that the spectral bias also holds in EA→B because it is a weighted sum over EB,i. In contrast, EbackA→B includes a constant term 2(1− ρ)η2i . This constant term causes catastrophic forgetting, as we show later.

heading: 4.1 TRANSITION FROM NEGATIVE TO POSITIVE TRANSFER
text: We now look at more detailed behaviors of generalization errors obtained in Theorem 1. We first discuss the role of the target similarity ρ for improving generalization. Figure 1(a) shows the comparison of the generalization between single-task training and sequential training. Solid lines show theory, and markers show experimental results of trained neural networks in the NTK regime. We trained the model (1) with ReLU activation, L = 3, and Ml = 4, 000 by using the gradient descent over 50 trials. More detailed settings of our experiments are summarized in Section E.2. Because we set NA = NB = 100, we have EA = EB . The point is that both EA→B(ρ) and EbackA→B(ρ) are lower than EA(= EB) for large ρ. This means that the sequential training degrades generalization if the targets are dissimilar, that is, negative transfer. In particular, EbackA→B(ρ) rapidly deteriorates for the dissimilarity of targets. Note that both EA→B and EbackA→B are linear functions of ρ. Figure 1(a) indicates that the latter has a large slope. We can gain quantitative insight into the critical value of ρ for the negative transfer as follows.
Knowledge transfer. The following asymptotic equation gives us the critical value for EA→B :
EA→B(ρ)/EB ∼ 2(1− ρ) for NA NB . (15)
A straightforward algebra leads to this (Section A.3). In the context of transfer learning, it is reasonable that the target domain has a limited sample size compared to the first task. For ρ > 1/2, we have EA→B < EB , that is, previous task A contributes to improving the generalization performance on subsequent task B (i.e., positive transfer). For ρ < 1/2, however, negative transfer appears.
The following sufficient condition for the negative transfer is also noteworthy. By evaluatingEA→B > EB , we can prove that for any NA and NB , the negative transfer always appears for
ρ < ρ∗ := √ γA/(1 + √ γA). (16)
This is just a sufficient condition and may be loose. For example, we asymptotically have the critical target similarity ρ = 1/2 > ρ∗ for NA NB . Nevertheless, this sufficient condition is attractive in the sense that it clarifies the unavoidable negative transfer for the small target similarity.
Backward transfer. The EbackA→B(0) includes the constant term ∑ i η 2 i independent of sample sizes. Note that qA and qB decrease to zero for large sample sizes (Bordelon et al., 2020), and we have EbackA→B(ρ) ∼ ∑ i η 2 i (1 − ρ). In contrast, EA converges to zero for a large NA. Therefore, the intersection between EbackA→B(ρ) and EA reach ρ = 1 as NA and NB increase. This means that when we have a sufficient number of training samples, negative backward transfer (EbackA→B(ρ) > EA) occurs even for very similar targets. Figure 1(a) confirms this result, and Figure 6 in Section A.6 shows more detailed learning curves of backward transfer. Catastrophic forgetting seems literally ”catastrophic” in the sense that the backward transfer rapidly deteriorates by the subtle target dissimilarity.

heading: 4.2 SELF-KNOWLEDGE TRANSFER
text: We have shown that target similarity is a key factor for knowledge transfer. We reveal that the sample size is another key factor. To clarify the role of sample sizes, we focus on the same target function (ρ = 1) in the following analysis. We refer to the knowledge transfer in this case as self-knowledge transfer to emphasize the network learning the same function by the same head. As is the same in ρ < 1, the knowledge obtained in the previous task is transferred as the network’s initialization for the subsequent training and determines the eventual performance.
Positive transfer by equal sample sizes. We find that positive transfer is guaranteed under equal sample sizes, that is, NA = NB . To characterize the advantage of the sequential training, we compare it with a model average: (fA + fB)/2, where fA (fB) means the model obtained by a single-task training on A (B). Note that since the model is a linear function of the parameter in the NTK regime, this average is equivalent to that of the trained parameters: (θA + θB)/2. After straightforward calculation in Section D, the generalization error of the model average is given by
Eave = (1− γB/2)EB . (17) Sequential training and model average include information of both tasks A and B; thus, it is interesting to compare it with EA→B . We find Proposition 3. For ρ = 1 and any NA = NB ,
EA→B(1) < Eave < EA = EB . (18)
The derivation is given in Section D. This proposition clarifies the superiority of sequential training over single-task training and even the average. The first task contributes to the improvement on the second task; thus, we have positive transfer.
Negative transfer by unbalanced sample sizes. While equal sample size leads to positive transfer, the following unbalanced sample sizes cause the negative transfer of self-knowledge:
EA→B(1)/EB ∼ 1/(1− γA) for NB NA. (19) The derivation is based on Jensen’s inequality (Section A.3). While EA→B and EB asymptotically decrease to zero for the large NB , their ratio remains finite. Because 0 < γA < 1, EA→B(1) > EB . It indicates that the small sample size of task A leads to a bad initialization of subsequent training and makes the training on task B hard to find a better solution.
Figure 1(b) summarizes the learning curves which depend on sample sizes in a highly non-linear way. Solid lines show theory, and markers show the results of NTK regression over 100 trials. The figure shows excellent agreement between theory and experiments. Although we have complicated transitions from positive to negative transfer, our theoretical analyses capture the basic characteristics of the learning curves. For self-knowledge transfer (ρ = 1), we can achieve positive transfer at NA/NB = 1, as shown in Proposition 3, and for large NA/NB , as shown in (15). In contrast, negative transfer appears for small NA/NB , as shown in (19). For ρ < 1, large NA/NB produces positive transfer for ρ < 1/2, as shown in (15).
If we set a relatively large σ > 0, the learning curve may become much more complicated due to multiple descent. Figure 5 in Section A.5 confirms the case in which multiple descent appears in EA→B . The curve shape is generally characterized by the interaction among target similarity, self-knowledge transfer depending on sample size, and multiple descent caused by the noise.
4.3 SELF-KNOWLEDGE FORGETTING
We have shown in Section 4.1 that backward transfer is likely to cause catastrophic forgetting for ρ < 1. We show that even for ρ = 1, sequential training causes forgetting. That is, the training on task B degrades generalization even though both tasks A and B learn the same target.
We have EbackA→B(1) = EA→B(1) by definition, and Proposition 3 tells us that EA→B < EA. Therefore, no forgetting appears for equal sample sizes. In contrast, we have
EA→B(1)/EA ∼ 1/(1− γB) for NA NB . (20) One can obtain this asymptotic equation in the same manner as (19) since EA→B(1) is a symmetric function in terms of indices A and B. Combining (20) with (15), we have EA < EA→B(1) EB . Sequential training is better than using only task B, but training only on the first task is the best. One can say that the model forgot the target by task B despite learning the same target. We call this self-knowledge forgetting. In
the context of continual learning, many studies have investigated the catastrophic forgetting caused by different heads (i.e., in the situation of incremental learning). Our results suggest that even the sequential training on the same task and the same head shows such forgetting. Intuitively speaking, the self-knowledge forgetting is caused by the limited sample size of task B. Note that we have EA→B(1) = ∑ i q 2 A,iEB,i/(1 − γA). The generalization error of single-task training on task B (EB,i) takes a large value for a small NB and this causes the deterioration of EA→B as well. Figure 2 confirms the self-knowledge forgetting in NTK regression. We set NA = NB as the red line and NB = 100 as the yellow line. The dashed line shows the point where forgetting appears.

heading: 5 LEARNING CURVES OF MANY TASKS
text: We can generalize the sequential training between two tasks to that of more tasks. For simplicity, let us focus on the self-knowledge transfer (ρ = 1) and equal sample sizes. Applying Lemma 2 recursively from task to task, we can evaluate generalization defined by En = 〈∫ dxp(x)(f̄(x)− fn(x))2 〉 D.
Theorem 4. Assume that (i) (Xn, yn) (n = 1, ...,K) are given by the same distribution Xn ∼ P (X) and target yn = ∑ i w̄iψ(Xn) + εn. (ii) sample sizes are equal: Nn = N . For n = 1, ...,K,
En+1 = 1
(1− γ)2 q̃>Qn−1q̃ +Rn+1σ2, Q := diag
( q2 ) + N
(1− γ)κ2 q̃q̃>, (21)
where qi = κ/(κ+ ηiN), q̃i = ηiq2i and diag(q 2) denotes a diagonal matrix whose entries are q2i . The noise term Rn is a positive constant. In the noise-less case (σ = 0), the learning curve shows monotonic decrease: En+1 ≤ En. If all eigenvalues are positive, we have
En+1 < En (n = 1, 2, ...). (22)
See Section B for details of derivation. The learning curve (i.e., generalization error to the number of tasks) monotonically decreases for the noise-less case. the monotonic decrease comes from λmax(Q) < 1. This result means that the self-knowledge is transferred and accumulated from task to task and contributes in improving generalization. It also ensures that no self-knowledge forgetting appears. We can also show that Rn converges to a positive constant term for a large n and the contribution of noise remains as a constant.
KRR-like expression. The main purpose of this work was to address the generalization of the continually trained model fn. As a side remark, we show another expression of fn:
[Θ (x′, X1) · · ·Θ (x′, Xn)]  Θ (X1) O · · · O Θ (X2, X1) Θ (X2) ...
... . . . O
Θ (Xn, X1) · · · Θ (Xn)
 −1  y1...
yn  . (23) This easily comes from comparing the update (4) with a formula for the inversion of triangular block matrices (Section C). One can see this expression as an approximation of KRR, where the upper triangular block of NTK is set to zero. Usual modification of KRR is diagonal, e.g., L2 regularization and block approximation, and it seems that the generalization error of this type of model has never been explored. Our result revealed that this model provides non-trivial sample size dependencies such as self-knowledge transfer and forgetting.

heading: 5.1 EXPERIMENTS
text: Although Theorem 4 guarantees the monotonic learning curve for equal sizes, unbalanced sample sizes should cause a non-monotonic learning curve. We empirically confirmed this below.
Synthetic data. First, we empirically confirm En on synthetic data (10). Figure 3(a1) confirms the claim of Theorem 4 that the generalization error monotonically decreases for σ = 0 as the task number increases. Dashed lines are theoretical values calculated using the theorem, and points with error bars were numerically obtained by fn (4) over 100 trials. For σ > 0, the decrease was suppressed. We set σ2 = {0, 10−5, 10−4, 10−3} and Ni = 100. Figure 3(a2) shows self-knowledge forgetting. When the first task has a large sample size, the generalization error by the second task can increase for small subsequent sample sizes Ni. For smaller Ni, there was a tendency for the error to keep increasing and taking higher errors than that of the first task during several tasks. In practice, one may face a situation where the model is initialized by the first task training on many samples and then trained in a continual learning manner under a memory constraint. The figure suggests that if the number of subsequent tasks is limited, we need only the training on the first task. If we have a sufficiently large number of tasks, generalization eventually improves.
MLP on MNIST / ResNet-18 on CIFAR-10. We mainly focus on the theoretical analysis in the NTK regime, but it will be interesting to investigate whether our results also hold in more practical settings of deep learning. We trained MLP (fully-connected neural networks with 4 hidden layers) and ResNet-18 with stochastic gradient descent (SGD) and cross-entropy loss. We set the number of epochs sufficient for the training error to converge to zero for each task. We confirmed that they show qualitatively similar results as in the NTK regime. We randomly divided the dataset into tasks without overlap of training samples. Figure 3(b1,c1) shows the monotonic decrease for an equal sample size and that the noise suppressed the decrease. We set Ni = 500 and generated the noise by the label corruption with a corruption probability {0, 0.2, ..., 0.8} (Zhang et al., 2017). The vertical axis means Error = 1 − (Test accuracy [%])/100. Figure 3(b2,c2) shows that unbalanced sample sizes caused the non-monotonic learning curve, similar to NTK regression.

heading: 6 DISCUSSION
text: We provided novel quantitative insight into knowledge transfer and forgetting in continual learning. Even in the NTK regime, where the model is simple and linearized, learning curves show rich and non-monotonic behaviors depending on both target similarity and sample size. Even in training on the same target, we revealed that self-knowledge transfer is successful or forgetting appears depending on the balance of sample sizes. This result indicates that the performance of continual learning is more complicated than we thought, but we can still find some universal laws behind it.
There are other research directions to be explored. While we focused on reporting novel phenomena on transfer and forgetting, it is also important to develop efficient algorithms to achieve better performance. To mitigate catastrophic forgetting, previous studies proposed several strategies such as regularization, parameter isolation, and experience replay (Mirzadeh et al., 2020). Evaluating such strategies with theoretical backing would be helpful for further development and improve the design of continual learning. For example, orthogonal projection methods modify gradient directions (Doan et al., 2021) and replay methods allow the reuse of the previous samples. We conjecture that these could be analyzed by extending our calculations in a relatively straightforward way. It would also be interesting to investigate richer but complicated situations required in practice, such as streaming of non-i.i.d. data and distribution shift (Aljundi et al., 2019). The current work and other theoretical works in continual learning or transfer learning basically assume the same input distribution between different tasks (Lee et al., 2021; Tripuraneni et al., 2020). Extending these theories to the case with an input distribution shift will be essential for some applications including domain incremental learning. Our analysis may also be extended to different topics with a similar setting as continual learning. For example, self-distillation uses the outputs of the model for training in the subsequent phase and plays an interesting role of regularization (Mobahi et al., 2020).
While our study provides universal results, which do not depend on specific eigenvalue spectra, it is interesting to investigate individual cases depending on architecture. Studies on NTK eigenvalues have made remarkable progress, covering shallow and deep ReLU neural networks (Geifman et al., 2020; Chen & Xu, 2020), skip connections (Belfer et al., 2021), and CNNs (Favero et al., 2021). We expect that our analysis and findings will serve as a foundation for further understanding and development on theory and experiments of continual learning.

heading: REPRODUCIBILITY STATEMENT
text: The main contributions of this work are theoretical claims, and we clearly explained their assumptions and settings in the main text. Complete proofs of the claims are given in the Supplementary Materials. For experimental results of training deep neural networks, we used only already-known models and algorithms implemented in Pytorch. All of the detailed settings, including learning procedures and hyperparameters, are clearly explained in Section E.2.

heading: Supplementary Materials

heading: A SEQUENTIAL TRAINING BETWEEN TWO TASKS

heading: A.1 NOTATIONS
text: The kernel ridge regression is given by
f∗ = arg min f∈H
1
2λ N∑ µ=1 (f(xµ)− yµ)2 + 1 2 〈f, f〉H (S.1)
where N is the sample size andH is the reproducing kernel Hilbert space (RKHS) induced by the neural tangent kernel Θ. We assume that the input samples xµ are generated from a probabilistic distribution p(x) and that the NTK has finite trace, i.e., ∫ dxp(x)Θ(x, x) <∞. Mercer’s decomposition of Θ is expressed by∫ dx′p (x′) Θ (x, x′)φi (x ′) = ηiφi(x) (i = 0, 1, . . . ,∞). (S.2) In other words, we have Θ(x′, x) = ∑∞ i=0 ηiφi(x
′)φi(x). Note that a function belonging to RKHS is given by
f(x) = ∞∑ i=0 aiφi(x) (S.3)
with ‖f‖2H = ∑∞ i=0 a 2 i /ηi < ∞. Then, using the orthonormal bases {φi}∞i=0, the solution of the regression is given by
f∗(x) = ∑ i w∗i ψi(x), w ∗ = ( ΨΨ> + λI )−1 Ψy (S.4)
where ψi(x) := √ ηiφi(x). Each column of Ψ is given by φ(xµ) (µ = 1, ..., N ). We focus on the NTK regime and take the ridge-less limit (λ→ 0). The sequentially trained model (4) between tasks A and B is written as
fA→B(x)− fA(x) = Θ(x,B)Θ(B)−1 (yB − fA(B)) , (S.5)
fA(x) = Θ(x,A)Θ(A) −1yA. (S.6)
The model fA is trained on single task A and represented by
fA(x) = w ∗> A ψ(x) (S.7)
with w∗A = lim
λ→0 argminwA HA(wA), (S.8)
HA(wA) := 1
2λ NA∑ µ=1 ( w>Aψ(x µ A)− y µ A )2 + 1 2 ‖wA‖22. (S.9)
Eq. (S.9 ) is the objective function equivalent to (S.1 ) because 〈fA, fA〉H = ‖wA‖22. Similarly, we can represent fA→B by the series expansion with the bases of RKHS. Note that the right-hand side of (S.5 ) is equivalent to the kernel ridge-less regression on task B’s input samples and labels yB − fA(B). We have
fA→B(x) = (w ∗ A + w ∗ B) >ψ(x) (S.10)
with w∗B = lim
λ→0 argminwB H(wB , w
∗ A), (S.11)
H(wB , w ∗ A) :=
1
2λ NB∑ µ=1 ( w>Bψ(x µ B)− (y µ B − w ∗> A ψ(x µ B)) )2 + 1 2 ‖wB‖22. (S.12)

heading: A.2 PROOF OF THEOREM 1.

heading: A.2.1 KNOWLEDGE TRANSFER EA→B
text: First, we take the average over training samples of task B conditioned by task A, that is,
EA→B|A := 〈∫ dxp(x)(f̄B(x)− (w∗A + w∗B)>ψ(x))2 〉 DB
(S.13)
= 〈∫ dxp(x)((w∗B − (w̄B − w∗A))>ψ(x))2 〉 DB
(S.14)
where the set of task B’s training samples is denoted by DB . We have
EA→B = 〈EA→B|A〉DA . (S.15)
Note that the objective function H(wB , w∗A) is transformed to
H(wB , w ∗ A) =
1
2λ NB∑ µ=1 ( wB − (w̄B − w∗A))>ψ(x (µ) B ) )2 + 1 2 ‖wB‖22. (S.16)
Comparing (S.14 ) and (S.16 ), one can see that the average over task B conditioned by task A is equivalent to single-task training with the target function (w̄B − w∗A)>ψ(x). Therefore, by using (8), we immediately obtain
EA→B|A = 1
1− γB ∑ i ηi(w̄B,i − w∗A,i)2 ( κB κB +NBηi )2 + γB 1− γB σ2 (S.17)
=: ∑ i φi(w̄B,i − w∗A,i)2 + γB 1− γB σ2. (S.18)
Next, we take the average of (S.18 ) over task A. Only w̄∗A depends on the task A and is determined by the single-task training (S.8 ). This corresponds to Lemma 2 with u = w̄B and φi = ηiq2B,i/(1−γB). We obtain
EA→B = ∑ i
[ (w̄A,i − w̄B,i)2 − 2w̄A,i(w̄A,i − w̄B,i)qA,i
+ w̄A,i + 1 1− γA ηiNA κ2A ∑ j ηjw̄ 2 A,jq 2 A,j  q2A,i ] ηiq 2 B,i 1− γB
+ σ2
( 1
1− γA 1 1− γB NA κ2A ∑ i η2i q 2 A,iq 2 B,i + γB 1− γB
) . (S.19)
Although this is a general result that holds for any w̄A and w̄B , it is a bit complicated and seems not easy to give an intuitive explanation. Let us take the average over
[w̄A,i, w̄B,i] ∼ N (0, ηi [ 1 ρ ρ 1 ] ). (S.20)
The generalization error is then simplified to
EA→B(ρ) = 〈EA→B〉w (S.21)
= ∑ i
[ 2(1− ρ) (1− qA,i) +
q2A,i 1− γA
] EB,i
+ σ2
( 1
1− γA 1 1− γB NA κ2A ∑ i η2i q 2 A,iq 2 B,i + γB 1− γB
) , (S.22)
where we used 〈w2A,i〉w = 〈w2B,i〉w = ηi and 〈wA,i, wB,i〉w = ρ.

heading: A.2.2 BACKWARD TRANSFER
text: Backward transfer is measured by the prediction on the previous task A:
EbackA→B := 〈∫ dxp(x)(f̄A(x)− fA→B(x))2 〉 (S.23)
= 〈∫ dxp(x)((w∗B − (w̄A − w∗A))>ψ(x))2 〉 . (S.24)
First, we take the average over task B,
EbackA→B|A = 〈∫ dxp(x)((w∗ − (w̄A − w∗A))>ψ(x))2 〉 DB . (S.25)
This corresponds to Lemma 2 with the replacement φi ← ηi and u ← w̄A − w∗A. Recall that the objective function of NTK regression was given by (S.16). The target w̄A in Lemma 2 is replaced as w̄A ← w̄B − w∗A. We then have
EbackA→B|A = ∑ i [ (w̄B,i − w̄A,i)2 − 2(w̄B,i − w∗A,i)(w̄B,i − w̄A,i)qB,i
+ (w̄B,i − w∗A,i)2 + 11− γB ηiNBκ2B ( ∑ j=0 ηj(w̄B,j − w∗A,j)2q2B,j + σ2)  q2B,i ] ηi
(S.26)
= γB ∑ i ηi(w̄B,i − w̄A,i)2 + ∑ i ηiq 2 B,i 1− γB ( wA,i − (w̄B,i − 1− γB qB,i (w̄B,i − w̄A,i)) )2
+ γB
1− γB σ2. (S.27)
Next, we take the average over DA. Since the first and third terms of (S.27 ) are independent of DA, we need to evaluate only the second term. The second term corresponds to Lemma 2 with φi = ηiq 2 B,i/(1− γB) and ui = w̄B,i − (1− γB)/qB,i(w̄B,i − w̄A,i). We obtain
EbackA→B = 〈 EbackA→B|A 〉 DA
= γB ∑ i ηi(w̄B,i − w̄A,i)2 + ∑ i
[ (w̄A,i − w̄B,i)2(qB,i − (1− γB))2
− 2w̄A,i(w̄A,i − w̄B,i)(qB,i − (1− γB))qA,iqB,i
+ w̄2A,i + 11− γA ηiNAκ2A ( ∑ j ηjw 2 A,jq 2 A,j + σ 2)  q2B,iq2B,i ] ηi 1− γB
+ σ2
( 1
1− γA 1 1− γB NA κ2A ∑ i η2i q 2 A,iq 2 B,i + γB 1− γB
) . (S.28)
Finally, by taking the average over (S.20 ), we have
EbackA→B(ρ) = 〈EbackA→B〉w = ∑ i [ 2γBη 2 i (1− ρ) + 2 (qB,i − (1− γB)) 2 η 2 i 1− γB (1− ρ)
− 2 η2i (qB,i − 1 + γB) qB,iqA,i 1− γB (1− ρ) + 1 1− γA 1 1− γB η2i q 2 A,iq 2 B,i ] + σ2 ( 1
1− γA 1 1− γB NA κ2A ∑ i η2i q 2 A,iq 2 B,i + γB 1− γB
)
= 2(1− ρ) ∑ i η2i
[ 1 + qB,i(qA,i − 2) +
q2B,i(1− qA,i) 1− γB
] +
1 1− γA 1 1− γB
× ∑ i η2i q 2 A,iq 2 B,i + σ 2
( 1
1− γA 1 1− γB NA κ2A ∑ i η2i q 2 A,iq 2 B,i + γB 1− γB
) . (S.29)
A.2.3 LEMMA 2 Lemma 2. Suppose training on single task A, the target of which is given by f̄(x) = ∑ i w̄A,iψi(x),
and denote the trained model (2) as f∗ = ∑ i w ∗ A,iψi. Define the following cost function:
E = 〈∑ i φi(w ∗ A,i − ui)2 〉 DA
(S.30)
for arbitrary constants φi and ui. Using the replica method under a sufficiently large NA, we have
E = ∑ i=0
[ (w̄A,i − ui)2 − 2w̄A,i(w̄A,i − ui)qA,i
+ w̄2A,i + 11− γA ηiNAκ2A ( ∑ j=0 ηjw̄ 2 A,jq 2 A,j + σ 2)  q2A,i ] φi. (S.31)
Proof. The process of derivation is similar to Canatar et al. (2021), but we have additional constants Φ and c. They cause several differences in the detailed form of equations.
Define
Z[J ] = ∫ dwA exp(−βHA(wA) + J βN
2 E(wA)) (S.32)
where E(wA) = 〈∑ i φi(wA,i − ui)2 〉 DA . We omit the index A in the following. We have
E = lim β→∞
2
βN
∂
∂J 〈logZ[J ]〉D ∣∣∣∣ J=0 . (S.33)
To evaluate 〈logZ〉, we use the replica method (a.k.a. replica trick):
〈logZ〉D = lim n→0
1 n (〈Zn〉D − 1) (S.34)
The point of the replica method is that we first calculate Zn for n ∈ N then take the limit of n to zero by treating it as a real number. In addition, we calculate the average 〈Zn〉D under a replica symmetric ansätz and a Gaussian approximation by following the calculation procedure of the previous works (Dietrich et al., 1999; Bordelon et al., 2020; Canatar et al., 2021).
We have 〈Zn〉D = ∫ dWn exp ( −β
2 n∑ a=1 ‖wa‖2 + βJN 2 n∑ a=1 (wa − u)>Φ (wa − u) ) 〈Q〉NA{xµ,εµ} ,
(S.35)
Q := exp ( − β
2λ n∑ a=1 ( (waA − w̄A)>ψ(xµ)− εµ
)2) , (S.36)
where we define dWn = ∏n a=1 dw
a and Φ is a diagonal matrix whose diagonal entries given by φi. We take the shift of wa → wa + w̄. Then,
〈Zn〉D = ∫ dWn exp(− nβ
2 ‖w̄‖2 + nβJN 2 (w̄ − u)>Φ(w̄ − u))︸ ︷︷ ︸
=:Z̄(J) · exp( ∑ a −β 2 ‖wa‖2 + βJN 2 wa>Φwa − βk>wa) 〈Q〉N{xµ,εµ} , (S.37)
Q = exp(− β 2λ n∑ a=1 (wa>ψ(xµ)− εµ)2), (S.38)
k̄ := w̄ − JNΦ(w̄ − u). (S.39)
First, let us calculate 〈Q〉{xµ,εµ}. This term is exactly the same as appeared in the previous works (Bordelon et al., 2020; Canatar et al., 2021). To describe notations, we overview their derivation. Define qa = wa>ψ (x) + ε, which are called order parameters. We approximate the probability distribution of qa by a multivariate Gaussian:
P ({qa}) = 1√ (2π)n det(C) exp −1 2 ∑ a,b (qa − µa) [C−1]ab ( qb − µb ) , (S.40) with
µa := 〈qa〉{x,ε} = 〈 wa>ψ(x) 〉 x + 〈ε〉ε = √ η0w a 0 , (S.41)
Cab := 〈 qaqb 〉 {x,ε} = w a>〈ψ(x)ψ(x)>〉xwb + 〈εaεb〉ε = wa>Λwa + Σab, (S.42)
where Σab = σδab, 〈· · · 〉x denotes the average over p(x) and Λ is a diagonal matrix whose entries are ηi. We have √ η0w a 0 in (S.41 ) since η0 corresponds to the constant shift φ0(x) = 1. Training samples are i.i.d. and we omitted the index µ. We then have
〈Q〉{x,ε} = exp
( −1
2 log det
( I + β
λ C
) − β 2λ µ> ( I + β λ C )−1 µ ) , (S.43)
where I denotes the identity matrix. Define conjugate variables {µ̂a, Ĉab} by the following identity: 1 = C ∫ ∏
a≥b
dµadµ̂adCabdĈab  × exp
−N∑ a µ̂a ( µa − waA,0 √ η0 ) −N ∑ a≥b Ĉab ( Cab − wa>Λwb − Σab ) , (S.44) where C denotes an uninteresting constant and we took the conjugate variables on imaginary axes. Next, we perform the integral over Wn. Eq. (S.37 ) becomes 〈Zn〉D
= CZ̄(J) ∫ dWn ∏ a≥b dΩab exp(−N( ∑ a µ̂aµa + ∑ a≥b Ĉab(Cab − Σab))) exp[−NG−GS ]
(S.45)
where dΩab = dµadµ̂adCabdĈab and define
G = 1
2 log det
( I + β
λ C
) + β 2λ µ> ( I + β λ C )−1 µ, (S.46)
exp(−GS) = exp( ∑ a (−β 2 ‖wa‖2 + βJN 2 wa>Φwa − βk>wa))
× exp N∑ a µ̂awaA,0 √ η0 +N ∑ a≥b Ĉabwa>Λwb  . (S.47) We can represent
∫ dWn exp(−GS) by∫ dWn exp(−GS) =
∏ i ∫ dxi exp ( −β 2 x>i Q̂ixi − βb>i xi ) (S.48)
where xi ∈ Rn denotes a vector [w1i , ..., wai , ..., wni ]> and i is the index of kernel’s eigenvalue mode (i = 0, 1, ...). We defined
Q̂i = (1− φiJN)In − ηiN
β (C̃ + diag(C̃)), (S.49)
bi = k̄i1n (i ≥ 1), (S.50)
b0 = k̄01n − N √ η0
β µ̂, (S.51)
where In is an n× n identity matrix and 1n is an n-dimensional vector whose all entries are 1. The GS term includes φ and c that are specific to our study. When φ = η and u = w̄, it is reduced to previous works (Bordelon et al., 2020; Canatar et al., 2021). Taking the integral over {xi}, we have∫
dWn exp(−GS) = C ∏ i=0 exp( β 2 b>i Q̂ −1 i bi)/ √ det Q̂i. (S.52)
That is,
GS = 1
2 ∑ i=0 log det Q̂i − β 2 ∑ i=0 b>i Q̂ −1 i bi. (S.53)

heading: REPLICA SYMMETRY AND SADDLE-POINT METHOD
text: Next, we carry out the integral (S.47 ) by the saddle-point method. Assume the replica symmetry:
µ = µa, r = Caa, c = Ca 6=b, (S.54)
µ̂ = µ̂a, r̂ = Ĉaa, ĉ = Ĉa 6=b. (S.55)
The following three terms are the same as in the previous works:
det ( I + β
λ C
) = [ 1 + β
λ (r − c)
]n( 1 + n
βc
λ+ β (r − c) ) , (S.56)(
I + β
λ C
)−1 =
1
1 + βλ (r − c)
( I − βc
λ+ β (r − c) + nβc 11>
) , (S.57)
µ̂>µ+ ∑ a≥b Ĉab(Cab − Σab) = n ( µ̂µ+ r̂(r − σ2)− 1 2 q̂(q − σ2) ) , (S.58)
where 11> denotes a matrix whose all entries are 1. Regarding the leading term of order n (n→ 0),
log det ( I + β
λ C
) = n log ( 1 + β
λ (r − c)
) + n
βC
λ+ β (r − c) , (S.59)
µ> ( I + β
λ C
)−1 µ =
nµ2
1 + βλ (r − c) . (S.60)
Furthermore. we have
Q̂i = (1− φiJN − ηiN
β (2r̂ − ĉ))In −
ηiNĉ
β 1n1 > n . (S.61)
After straightforward algebra, the leading terms become
log det Q̂i = n ln gi − n ηiNĉ
giβ , (S.62)
1>n Q̂ −1 i 1n =
n gi , (S.63)
with
gi := 1− φiJN − ηiN
β (2r̂ − ĉ). (S.64)
Substituting these leading terms into (S.45 ), we obtain 〈Zn〉D = C ∫ dθ exp(−nNS(θ)) with
S(θ) = −βJ 2 (w̄ − u)>Φ(w̄ − u)
= ( µ̂µ+ r̂(r − σ2)− 1
2 ĉ(c− σ2)
) + 1
2
( log β (r − c) + c+ µ 2
r − c ) + 1
2N ∑ i=0 (ln gi − ( ηiNĉ β + βk̄2i )/gi)− 1 2g0 (−2k̄0µ̂ √ η0 +Nη0µ̂ 2/β), (S.65)
where θ denotes a set of variables {r, r̂, c, ĉ, µ, µ̂}. Here, we take N 1 and use the saddle-point method. We calculate saddle-point equations ∂S(θ∗)/∂θ = 0 and obtain
µ̂ = − µ r − c , (S.66)
r̂ = 1
2
( c+ µ2
(r − c)2 − 1 r − c
) , (S.67)
ĉ = c+ µ2
(r − c)2 , (S.68)
µ = Nη0µ̂ g0β − k̄0 g0 √ η0, (S.69)
r = ∑ i=0 ( ηiNĉ β + βk̄2i ) ηi g2i β + η0N g20β (−2k̄0µ̂ √ η0 +Nη0µ̂ 2β) + σ2, (S.70)
c = ∑ i=0 ( ηiNĉ β + βk̄2i ) ηi g2i β − 1 β ∑ i=0 ηi gi + η0N g20β (−2k̄0µ̂ √ η0 +Nη0µ̂ 2β) + σ2. (S.71)
From (S.67 ) and (S.68 ),
r − c = − 1 2r̂ − ĉ . (S.72)
From (S.70 ) and (S.71 ), we also have
β(r − c) = ∑ i=0 ηi gi =: κ. (S.73)
Substituting (S.64 ) and (S.72 ) into (S.73 ), We obtain the implicit function for κ:
1 = ∑ i=0 ηi W (φi) , W (φi) := κ(1− φiJN) + ηiN. (S.74)
Using κ and W , the saddle-point equations become
µ̂ = −βµ κ , (S.75)
r̂ = 1
2
( c+ µ2
κ2 β2 − β κ
) , (S.76)
ĉ = β2
κ2 (c+ µ2), (S.77)
µ = − √ η0k̄0κ
Nη0 +W (φ0) , (S.78)
r = c+ κ
β . (S.79)
Substituting back these quantities into (S.65 ), we obtain
S(θ∗) = −βJ 2 (w̄ − u)>Φ(w̄ − u)1 2 lnκ+ 1 2N ∑ i=0 ln gi − β 2N ∑ i=0 k̄2i gi
+ βη0κ
2W (φ0) k̄20 Nη0 +W (φ0) + β 2κ σ2 + const. (S.80)
Recall 〈Zn〉D ≈ exp(−nNS(θ∗)). We have 〈logZ〉D = −NS(θ∗). (S.81)

heading: GENERALIZATION ERROR
text: We evaluate
E = lim β→∞
2
βN
∂
∂J 〈logZ〉D ∣∣∣∣ J=0 = − lim β→∞ 2 β ∂ ∂J S(θ∗) ∣∣∣∣ J=0 . (S.82)
Note that W, g, κ and k̄ depend on J . At the point of J = 0, ∂W (φi)
∂J = −φiNκ+
∂κ ∂J , (S.83)
∂gi ∂J = −(φi + 1 κ2 ∂κ ∂J ηi)N, (S.84)
∂κ ∂J =
κ2N 1− γ ∑ i=0 ηiφi (κ+ ηiN)2 , (S.85)
∂k̄i ∂J
= −Nφi(w̄i − ui). (S.86) Substituting (S.83 -S.86 ) into (S.82 ), we obtain E =∑ i=0 [ (w̄i − ui)2 − 2w̄i(w̄i − ui)qi + w̄2i + 11− γ ηiNκ2 (∑ j=0 ηjw̄ 2 j q 2 j + σ 2)  q2i ] φi + ξ0,
(S.87) where ξ0 is
ξ0 = η0w̄ 2 0
[( κ
Nη0 + 2W0 W 20 (Nη0 +W0) 2 − 1 W0(Nη0 +W0)
)( N
1− γ ∑ i ηiq 2 i φi
)
− φ0Nκ2 Nη0 + 2W0
W 20 (Nη0 +W0) 2
] + 2η0
φ0κNw̄0(w̄0 − u0) W0(Nη0 +W0)
(S.88)
with W0 = κ+Nη0. When η0 = 0, we have ξ0 = 0. Note that the additional term ξ0 is not specific to our case but also appeared in the previous work (Canatar et al., 2021). We have ξ0 = O(1/η0) for a large η0 and ξ0 is often negligibly small.

heading: A.3 EQUATIONS FOR UNDERSTANDING NEGATIVE TRANSFER

heading: A.3.1 DERIVATION OF EQUATION 15
text: First, let us evaluate the term including qA,i = κA/(κA +NAηi). Note that κA is a monotonically decreasing function of NA because
∂κA ∂NA = − (∑ i ηi (κA +NAηi)2 )−1∑ i η2i (κA +NAηi)2 < 0, (S.89)
which comes from the implicit function theorem. Let us write κA at a finite NA = c as κA(c). We then have
qA,i ≤ κA(c)
NAηi , (S.90)
for NA > c. Next, we evaluate EA→B(ρ) = 2(1− ρ)EB − 2(1− ρ) ∑ i qA,iEB,i + ∑ i q2A,i 1− γA EB,i. (S.91)
The second term is negligibly small for a sufficiently large NA > c because∑ i qA,iEB,i ≤ κA(c) NA ∑ i ηiq 2 B,i 1− γB = κA(c) NA κB . (S.92)
Note that we have 1− γ = κ ∑ i=0 ηi/(κ+ ηiN)
2. The third term is∑ i q2A,i 1− γA EB,i ≤ EA 1− γB = 1 1− γB γA 1− γA κ2A NA , (S.93)
where we used qB,i ≤ 1. It decreases to zero as NA increases. Thus, we have EA→B(ρ) ∼ 2(1− ρ)EB .

heading: A.3.2 DERIVATION OF EQUATION 19
text: EA→B(1)
EB =
1
1− γA
∑ i q 2 A,iq 2 B,iη
2 i∑
i q 2 B,iη 2 i
(S.94)
≥ 1 1− γA
( κA
κA + ∑ i q 2 B,iη 3 iNA
)2 (S.95)
where the second line comes from Jensen’s inequality. Using (S.90 ), we have∑ i q2B,iη 3 i ≤ κB(c) 2 N2B ∑ i ηi, (S.96)
for NB ≥ c. Since we assumed the finite trance of NTK (i.e., ∑ i ηi < ∞), the left-hand side of (S.96 ) is finite and converges to zero for a sufficiently large NB . Therefore, we have EA→B(1)
EB &
1
1− γA . (S.97)
In contrast, EA→B/EB ≤ 1/(1− γA) since qA ≤ 1. Thus, the ratio is sandwiched by 1/(1− γA).

heading: A.3.3 SUFFICIENT CONDITION FOR THE NEGATIVE TRANSFER
text: Let us denote the i-th mode of EA→B as EA→B,i and that of EB as EB,i. From EA→B,i > EB,i, we have
f(qi) := 2(1− ρ)(1− qi) + q2i
1− γ − 1 > 0, (S.98)
where f(q) is a quadratic function and takes the minimum at q∗ = (1− ρ)(1− γ). The condition f(q) > 0 holds if and only if
f(q∗) = −(1− γ)ρ2 − 2γρ+ γ > 0. (S.99) Solving this, we find ρ < √ γ/(1 + √ γ).

heading: A.4 PROOF OF PROPOSITION 3.
text: For NA = NB , we have qA,i = qB,i = qi. Then,
EA→B(1) = 1 (1− γ)2 ∑ i q4i η 2 i (S.100)
and the generalization error of single-task training is given by
E = 1 1− γ ∑ i q2i η 2 i . (S.101)
Model average is obtained in Section D and we have Eave = (1− γ/2)E < EB because 0 < γ < 1. Thus, we only need to evaluate the relation between Eave and EA→B :
Eave − EA→B = (1− γ
2 )
1 1− γ ∑ i q2i η 2 i −
1 (1− γ)2 ∑ i η2i q 4 i (S.102)
= (1− γ 2 ) 1 1− γ κ2 ∑ i a2i − κ2 (1− γ)2 ∑ i a2i q 2 i (S.103)
= κ2
(1− γ)2N {∑ i N2a3i (2−Nai)− 1 2 γ2(3− γ) } ︸ ︷︷ ︸
=:F
(S.104)
where we defined ai :=
ηi κ+ ηiN , (S.105)
and used qi = 1−aiN and γ = N ∑ i a 2 i . We can provide a lower bound of F by using the following cubic function: G(ai) := N
2a2i (2−Nai). We have 0 ≤ ai ≤ 1/N by definition. Let us consider a lower bound of G by using its tangent line at ai = t/N (0 ≤ t ≤ 1), that is, Nt(4− 3t)ai − 2t2(1− t). Define
H(ai) := G(ai)− (Nt(4− 3t)ai − 2t2(1− t)). We have H(0) = 2t2(1 − t) ≥ 0. Since H(1/N) = −2(t − 1)2(t − 1/2), we need t ≤ 1/2 to guarantee H(ai) ≥ 0 for all 0 ≤ ai ≤ 1/N . Here, note that H is a cubic function of ai and has two fixed points ai = t/N and (4− 3t)/(3N). We can see that for t ≤ 1/2, H has the local minimum at ai = t/N and
H(t/N) = t2(2− t)− (t2(4− 3t)− 2t2(1− t)) = 0. (S.106)
Therefore, we have H(ai) ≥ 0 for all 0 ≤ ai ≤ 1/N when the fixed constant t satisfies t ≤ 1/2. Thus, we have the lower bound of G:
G(ai) ≥ Nt(4− 3t)ai − 2t2(1− t). (S.107) Using this lower bound, we have
F = ∑ i aiG(ai)− 1 2 γ2(3− γ)
≥ γt(4− 3t)− 2t2(1− t)− 1 2 γ2(3− γ), (S.108)
where we used γ = N ∑ i a 2 i and ∑ i ai = 1. By setting t = γ/2, we obtain
F ≥ γ 2 2 (4− 3γ/2)− 2γ 2 4 (1− γ/2)− 1 2 γ2(3− γ)
= 0. (S.109) Therefore, we have Eave > EA→B .
A.5 MULTIPLE DESCENT
We overview the multiple descent of E1 investigated in Canatar et al. (2021). It appears for σ > 0. Let us set NB = αDl (α > 0, l ∈ N, D 1), which is called the l-th learning stage. We have EB,i = 0 (i < l), EB,l(α) and η2i (i > l). EB,l(α) is a function of α, and becomes a one-peak curve depending on the noise and the decay of kernel’s eigenvalue spectrum. Because each learning stage has a one-peak curve, the whole learning curve shows multiple descents as the sample size increases. Roughly speaking, the appearance of the multiple descent is controlled by γ. The γ is determined by the kernel’s spectrum and sample size N . For example, previous studies showed that if we assume the l-th learning stage, γ is a non-monotonic function of α, the maximum of which is γ = 1/( √ λ̄l + √ λ̄l + 1) 2 for a constant λ̄l = ∑ k>l η̄k/η̄l,
where η̄ is a normalized eigenvalue of the kernel (Canatar et al., 2021). This tells us that γ repeat the increase and decrease depending on the increase of the learning stage. Roughly speaking, this non-monotonic change of γ leads to the multiple descent.
In sequential training, EB,l(α) can similarly cause multiple descent as in single tasks because EA→B is a weighted summation over EB,i. Figure 5 is an example in which noise causes multiple decreases and increases depending on the sample sizes. We set ρ = 1 and obtained the theoretical values by (S.22 ). While EA→B(1) is a symmetric function of indices A and B for σ = 0, it is not for σ > 0. Depending on NA and NB , the learning ”surface” becomes much more complicated than the learning curve of a single task.

heading: A.6 ADDITIONAL EXPERIMENT ON BACKWARD TRANSFER
text: Figure 6 shows the learning curves of backward transfer. Solid lines show theory, and markers show the mean and interquartile range of NTK regression over 100 trials. The figure shows excellent agreement between theory and experiments. As is shown in Section 4.1, subtle target dissimilarity (ρ < 1) causes negative backward transfer (i.e., catastrophic forgetting). For a large NA, EbackA→B(ρ < 1) approaches a non-zero constant while EA approaches zero. Thus, we have EbackA→B > EA even for the target similarity close to 1. When sample sizes are unbalanced (NA NB), the learning curve shows negative transfer even for ρ = 1. This is the self-knowledge forgetting revealed in (20). The ratio EbackA→B(1)/EA takes a constant larger than 1, that is, 1/(1− γB).

heading: B PROOF OF THEOREM 4.

heading: B.1 LEARNING CURVE OF MANY TASKS
text: Eq. (4) is written as
fn(x) = n∑ k=1 Θ(x,Xk)Θ(Xk) −1(yk − fk−1(Xk)). (S.110)
Each term of this summation is equivalent to the kernel ridge-less regression on input samples Xk and labels yk − fk−1(Xk). Therefore, we can represent fn by
fn(x) = n∑ k=1 w∗>k ψ(x) (S.111)
with the minimization of the objective function H: w∗n = lim
λ→0 argminwn H(wn, w
∗ 1...(n−1)), (S.112)
H(wn, w ∗ 1...(n−1)) =
1
2λ Nn∑ µ=1 ( w>n ψ(x µ)− (yµn − n−1∑ k=1 w∗>k ψ(x µ)) )2 + 1 2 ‖wn‖22 (S.113)
= 1
2λ Nn∑ µ=1
( (wn − (w̄ −
n−1∑ k=1 w∗k)) >ψ(xµ)− εµn
)2 + 1
2 ‖wn‖22. (S.114)
The generalization error is
En+1 := 〈∫ dxp(x)(f̄(x)− fn+1(x))2 〉 D
(S.115)
= 〈∫ dxp(x)((w∗n+1 − (w̄ − n∑ k=1 w∗k)) >ψ(x))2 〉 D
(S.116)
where D = {D1, ...,Dn}. Since the data samples are independent of each other among different tasks, we can apply Lemma 2 sequentially from wn+1 to w1. First, we take the average over the (n+ 1)-th task, that is,
En+1|1:n = 〈∫ dxp(x)((w∗n+1 − (w̄ − n∑ k=1 w∗k)) >ψ(x))2 〉 Dn+1
(S.117)
This corresponds to Lemma 2 with φ = η and u = w̄A = w̄ − ∑n k=1 w ∗ k. We have
En+1|1:n = ∑ i=1 φ1,i(w̄i − n∑ k=1 w∗k,i) 2 +R1σ 2 (S.118)
with
φ1,i := ηiq
2 n+1,i
1− γn+1 , R1 := γn+1 1− γn+1 . (S.119)
Here, κn, γn and qn,k are determined by Nn. Next, we take the average over the n-th task: En+1|1:(n−1) = 〈 En+1|1:n 〉 Dn,εn (S.120)
= 〈∑ i φ1,i(w ∗ n,i − (w̄i − n−1∑ k=1 w∗k,i)) 2 〉 Dn +R1σ 2 (S.121)
This corresponds to Lemma 2 with φ = φ1 and u = w̄A = w̄ − ∑n−1 k=1 w ∗ k. We obtain
En+1|1:(n−1) = ∑ i=1 φ2,i(w̄ − n−1∑ k=1 w∗k) 2 +R2σ 2 (S.122)
with
φ2,i = φ1,iq 2 n,i + Nn κ2n(1− γn) ηiq 2 n,i ∑ j ηjφ1,jq 2 n,j , (S.123)
R2 = R1 + Nn κ2n(1− γn) ∑ j ηjφ1,jq 2 n,j . (S.124)
Similarly, we can take the averages from the (n− 1)-th task to the first task, and obtain En+1 = ∑ i=1 φn+1,iw̄ 2 i +Rn+1σ 2, (S.125)
φm+1,i = φm,iq 2 n−m+1,i + Nn−m+1 κ2n−m+1(1− γn−m+1) ηiq 2 n−m+1,i ∑ j ηjφm,jq 2 n−m+1,j  , (S.126)
for m = 1, ..., n, and
Rn+1 = n∑ m=1 Nn−m+1 κ2n−m+1(1− γn−m+1) ∑ j ηjφm,jq 2 n−m+1,j . (S.127)
They are general results for anyNn. By settingNn = N for all n, we can obtain a simpler formulation. In a vector representation, φn+1 is explicitly written as
φn+1 = 1
1− γ
[ diag(q2) +
N
(1− γ)κ2 q̃q̃>
]n q̃. (S.128)
Finally, by taking the average over w̄i ∼ N (0, ηi) which is the one-dimensional case of (S.20 ), we have
En+1 = 1 (1− γ)2 q̃> [ diag(q2) + N (1− γ)κ2 q̃q̃> ]n−1 q̃ +Rn+1σ 2 (S.129)
with
Rn+1 = N κ2(1− γ)2 n∑
m=1
q̃> [ diag(q2) + N
(1− γ)κ2 q̃q̃>
]m−1 q̃ + γ
1− γ . (S.130)
B.2 MONOTONIC DECREASE OF En
For σ2 = 0, we have
En+1 = 1
(1− γ)2 q̃>Qn−1q̃. (S.131)
Note that Q is a positive semi-definite matrix. If the maximum eigenvalue is no greater than 1, En is monotonically non-increasing with n. We denote the infinite norm as ‖A‖∞ = maxi ∑ j |Aij |. Because the infinite norm bounds the eigenvalues, we have
λ1(Q) ≤ max i ∑ j |Qij | (S.132)
= max i
( 1 + N
κ ηi
)( κ
κ+ ηiN )2 ︸ ︷︷ ︸
:=G(ηi)
, (S.133)
where G(η) is monotonically decreasing for η ≥ 0 and G(0) = 1. Therefore, the largest eigenvalue λ1(Q) is upper-bounded by 1, and En becomes monotonically decreasing. In particular. if ηi > 0 for all i, we have λ1(Q) < 1 and obtain En+1 < En. For λ1(Q) < 1, it is also easy to check that the series (S.130 ) in Rn+1 converges to a constant for large n.

heading: C DERIVATION OF KRR-LIKE EXPRESSION
text: KRR-like expression comes from the inverse formula of a block triangular matrix. Let us write (23) as
fn(x ′) = [Θ(x′, 1 : (n− 1)) Θ(x′, n)]Z−1n [ y1:(n−1) yn ] , (S.134)
where Zn is a block triangular matrix recursively defined by
Zn =
[ Zn−1 O
Θ(n, 1 : (n− 1)) Θ(n)
] . (S.135)
Then, (S.134 ) is transformed to [Θ(x′, 1 : (n− 1)) Θ(x′, n)] [
Z−1n−1 O −Θ(n)−1Θ(n, 1 : (n− 1))Z−1n−1 Θ(n)−1 ] [ y1:(n−1) yn ] (S.136)
= Θ(x′, 1 : (n− 1))Z−1n−1y1:(n−1) + Θ(x′, n)Θ(n)−1(yn −Θ(n, 1 : (n− 1))Z −1 n−1y1:(n−1))
(S.137)
= fn−1(x ′) + Θ(x′, n)Θ(n)−1(yn − fn−1(Xn)). (S.138)
Thus, one can see that the KRR-like expression is equivalent to our continually trained model.

heading: D DERIVATION OF MODEL AVERAGE
text: For clarity, we set σ = 0, NA = NB , and ρ = 1 (that is, w̄A = w̄B). Generalization error of a model average is expressed by
Eave =
〈∫ dxp(x) ( f̄B(x)− fA(x) + fB(x)
2
)2〉 D
(S.139)
= 〈∫ dxp(x) (( w̄B −
wA + wB 2
)> ψ(x) )2〉 D
(S.140)
= 〈( w̄B −
wA + wB 2
)> Λ ( w̄B −
wA + wB 2 )〉 D
(S.141)
We can evaluate Eave in a similar way to EA→B . First, take the average over DB . We define
Eave|A = 1
4
〈 ((wB − (2w̄B − wA))>Λ(wB − (2w̄B − wA)) 〉 DB . (S.142)
This average corresponds to Lemma 2 with replacement from the target task A to B, φ ← η and u← 2w̄B − wA. Then, we have
Eave|A = 1
4 ∑ i=0 { (wA,i − w̄B,i)2 − 2w̄B,i(wA,i − w̄B,i)qB,i + ( w̄2B,i + ηiNB κ2B EB ) q2B,i } ηi
(S.143)
= 1
4 ∑ i=0 (wA,i − (1 + qB,i)w̄B,i)2ηi + γB 4 EB . (S.144)
Next, we take the average over DA. The first term of (S.144 ) corresponds to Lemma 2 with replacement φ← η and u← (1 + qB,i)w̄B . Then, we get
Eave = 〈Eave|A〉DA (S.145)
= 1
4 ∑ i=0 ((1− qA,i)w̄A,i − (1 + qB,i)w̄B,i)2ηi + γA 4 EA + γB 4 EB . (S.146)
For NA = NB , we have qA = qB , γA = γB , and EA = EB . In addition, since we focused on w̄A = w̄B , we have
Eave = ∑ i=0 ηiw̄ 2 B,iq 2 B,i + γB 2 EB (S.147)
= (
1− γB 2
) EB (S.148)

heading: E EXPERIMENTAL SETTINGS

heading: E.1 DUAL REPRESENTATION
text: In dual representation, the targets (10) are given by [αA,i, αB,i] ∼ N (0, [ 1 ρ ρ 1 ] /P ′), (S.149)
f̄A(x) = P ′∑ j αA,jΘ(x ′ j , x), f̄B(x) = P ′∑ j αB,jΘ(x ′ j , x) (S.150)
for a sufficiently large P ′. We sampled x′ from the same input distribution p(x) and set P ′ = 104 in all experiments. It is easy to check that targets (10) and (S.150 ) are asymptotically equivalent for a large P ′. Because Θ(x′, x) = ∑ i ψi(x ′)ψi(x), we have ∑P ′ j αA,jΘ(x
′ j , x) =∑
i ∑P ′ j (αA,jψi(x ′ j))ψi(x) ∼ ∑ i w̄iψi(x), where w̄i is sampled from N (0, ηi).

heading: E.2 SUMMARIES ON EXPERIMENTAL SETUP
text: Figure 1(a). We trained the deep neural network (1) with ReLU, L = 3, and Ml = 4, 000 by the gradient descent. We set a learning rate 0.5 and trained the network during 10,000 steps over 50 trials with different random seeds. The target is given by (S.150 ) with C = 1, D = 10, and Gaussian input samples xi ∼ N (0, 1). We initialized the network with (σ2w, σ2b ) = (2, 0), set NA = NB = 100, and calculated the generalization error over 4, 000 samples. Each marker shows the mean and interquartile range over the trials. To calculate the theoretical curves, we need eigenvalues ηi of the NTK. We numerically compute them by the Gauss-Gegenbauer quadrature following the instruction of Bordelon et al. (2020). Its implementation is also given by Canatar et al. (2021). Since the input
samples are defined on a hyper-sphere Sd−1 and the NTK is a dot product kernel (i.e., Θ(x′, x) can be represented by Θ(x′>x)), the NTK can be decomposed into Gegenbauer polynomials:
Θ(z) = ∞∑ i=0 ηiN(d, i)Qi(z), (S.151)
where {Qi} are the Gegenbauer polynomials and N(d, i) are constants depending d and i. Since the Gegenbauer polynomials are orthogonal polynomials, we have
ηi = c ∫ 1 −1 Θ(z)Qi(z)dτ(z), (S.152)
for a certain constant c and dτ(z) = (1− z2)(d−3)/2dz. The Gauss-Gegenbaur quadrature approximates this integral by
ηi ≈ c r∑ j=1 wjΘ(zj)Qi(zj), (S.153)
where wj are constant weights and zj are the r roots of Qr(z). The definitions of constants (N(d, i), c and wj) are given in Section 8 of Bordelon et al. (2020) and we set r = 1000 as is the same in this previous work. We computed ηi (i = 1, .., 1000) and substituted them to the analytical expressions.
Figure 1(b). This figure shows the results of NTK regression; (2) for the single task and (4) for sequential training. We set NB = 4, 000, D = 50 and other settings were the same as in Figure 1(a).
Figure 2. This figure shows the results of NTK regression; we set D = 20, and other settings are the same as Figure 1(a).
Figure 3. (NTK regression) This figure shows the results of NTK regression; We set D = 10 and show the means and error bars over 100 trials. Other settings were the same as in Figure 1(a).
(MLP) We trained the deep neural network (1) with ReLU, L = 5, and Ml = 512 by SGD with a learning rate 0.001, momentum 0.9, and mini-batch size 32 over 10 trials with Pytorch seeds 1-10. We set the number of epochs to 150 and scaled the learning rate ×1/10 at the 100-th epoch, confirming that the training accuracy reached 1 for each task.
(ResNet-18) We trained ResNet-18 by SGD with a learning rate 0.01, momentum 0.9, and mini-batch size 128 over 10 trials with Pytorch seeds 1-10. We set the number of epochs to 150 and scaled the learning rate ×1/10 at the 100-th epoch, confirming that the training accuracy reached 1 for each task.
Note that the purpose of these experiments was not to achieve performance comparable to state-ofthe-art but to confirm self-knowledge transfer and forgetting. Therefore, we did not apply any data augmentation or regularization method such as weight decay and dropout.

Review summaries:
This paper provides a theoretical analysis for sequential training based on related NTK results.  It shows the similarity between the target functions is a key factor for forward and backward transfer, and when the target functions are the same the samples size of each task is a key factor to forward and backward transfer. It also shows even a slight difference between the target functions can cause catastrophic forgetting and the forgetting might still happen for the same target function when the sample size of a later task increases. 

This paper studies the generalization/transfer behavior of the continual learning/sequential learning under the neural tangent kernel (NTK) regime. It gives the formulation of the generalization error between two tasks in forwarding and backward ways and analyzes the influences of target similarity and sample sizes. Then the self-knowledge transfer case with the same target function is studied, which shows the universality of the catastrophic forgetting and the influence of the sample size. The results are then generalized to many tasks. 

The paper is a theoretical paper about continual learning, that studies 2 tasks settings in the NTK regime.  The paper study in particular transfer from tasks with the same target functions and introduce the concept of "self-knowledge transfer", i.e. transfer from two tasks with the same target function.

The paper presents a study of continual learning of neural networks in the neural tangent kernel (NTK) regime. The authors extend recent work which estimates the generalization error of such networks using a statistical mechanics technique known as the replica trick. The theory makes a number of predictions regarding backward and forward transfer learning which describe well the behavior of finite neural networks trained with gradient descent.