Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning
Authors: Mohammadamin Banayeeanzade
Mahdi Soltanolkotabi
Mohammad Rostami
Venue: Accepted by TMLR ()

Abstract: Multi-task learning (MTL) is a machine learning paradigm that aims to improve the generalization performance of a model on multiple related tasks by training it simultaneously on those tasks. Unlike MTL, where the model has instant access to the training data of all tasks, continual learning (CL) involves adapting to new sequentially arriving tasks over time without forgetting the previously acquired knowledge. Despite the wide practical adoption of CL and MTL and extensive literature on both areas, there remains a gap in the theoretical understanding of these methods when used with overparameterized models such as deep neural networks. This paper studies the overparameterized linear models as a proxy for more complex models. We develop theoretical results describing the effect of various system parameters on the model's performance in an MTL setup. Specifically, we study the impact of model size, dataset size, and task similarity on the generalization error and knowledge transfer. Additionally, we present theoretical results to characterize the performance of replay-based CL models. Our results reveal the impact of buffer size and model capacity on the forgetting rate in a CL setup and help shed light on some of the state-of-the-art CL methods. Finally, through extensive empirical evaluations, we demonstrate that our theoretical findings are also applicable to deep neural networks, offering valuable guidance for designing MTL and CL models in practice.

Published in Transactions on Machine Learning Research (03/2025)
Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning
Amin Banayeeanzade Deportment of Computer Science University of Southern California
banayeen@usc.edi
Mahdi Soltanolkotabi
soltamol@uusc.cm edu
Department Electrical and Computer Engineering University of Southern California
Mohammad Rostami
rostamim Wusu. edu
Department of Computer Science University of Souteenn California
Reviewed on OpenReview: https:// opwrreiiee.........o net/ forum? 42GPTOZumU
Abstract
Multi-task learning (MTT) is machine learning paradigm that aims to improve the gen- eralization performance of model multiple related tasks by training it simultaneously on those tasks. Unlike MTL, where the model has instant access to the training data of all tasks, continual learning (CL) involves adapting to new sequentially arriving tasks over time without forgetting the previously acquired knowledge. Despite the wide practical adoption of CL and MTL and extensive literature on both areas, there remains a gap in the theo- retical understanding of these methods when used with overparameterized models such as deep neural networks, This paper studies the overparameterized linear models as proxy for more complex models. We develop theoretical results describing the effect of various system parameters on the model's performance in MTT. setup. Specifically, study the impact of model size, dataset size, and task similarity on the generalization error and knowl- edge transfer. Additionally, we present theoretical results to characterize the performance of replay-based CL models. Our results reveal the impact buffer size and model capacity on the forgetting rate a CL setup and help shed light on some of the state-of-the-art CL methods. Finally, through extensive empirical evaluations, we demonstrate that our theoretical findings are also applicable to deep neural networks, offering valuable guidance for designing MTL and CL models in practice
1 Introduction
The dominant machine learning paradigm involves training single model on a single task and deploying the task-speeific model for the targeted application. This approach, while effective for many applications, often results in a highly specialized model that lacks the ability to generalize across different yet related tasks. To overcome this issue, Multi-Task Learning (MTL) methods train a single model on multiple tasks simultaneously with the goal of benefiting from the similarities between a collection of related tasks improve the efficiency of learning (Caruana) 1997). MTL not only enhances the performance of individual tasks but also facilitates the development of models that are more adaptable and capable of transferring knowledge from one task to another. This transferability is particularly valuable in real-world applications of deep learning where data can be scarce or imbalanced Zhang & Yang 2022).
Code available at http:://tthhb.cccmm/mmiiaaa//// Theory

---

Published in Transactions on Machine Learning Research (03/2025)
In many real-world applications, the tasks are not available for offline training, and instead, the agent needs to learn new tasks sequentially as they are encountered. Learning in these settings is particularly challenging due to the phenomenon known as catastrophic forgetting, where learning new tasks can lead to a significant loss of previously acquired knowledge (French, 1999). Continual Learning (CL) is a solution to this challenge that builds on the foundation of MTL but with a focus on the model's ability to learn continuously over time without forgetting (Parisi et al. 2019).
Despite the practical success of integrating MTL and CL with deep learning, there still remains critical need for theoretical understanding of learning mechanisms in these methods { have been previously made to thcoretically understand MTL (Baxter 2000 Ben-David & Borbely 2008). However, these endeavors are inapplicable to contemporary deep neural networks (DNN), where the models are heavily parameterized. In the overparameterized regime, DNNs exhibit peculiar generalization behav- iors, where despite having more parameters than training samples, they can still generalize well to unseen test data (Zhang et al.) 2017]. An important unexplored area, however, the effect of overparametriza- tion on the generalizabiilty of MTL models. More importantly, it is crucial to understand the impact of overparametrization on the possibility of effective cross-task knowledge transfer.
(Crawshaw 2020). Several efforts
DNN models optimized with SGD inherit some specific behaviors of linear models, specifically in overpa- rameterized regimes (Chizat et al. 2020). As a result, linear models have been widely studied as the first step toward understanding the deep double descent and benign overfitting (Hastie et al. 2022). However, previous investigations are limited to single-tast learning paradigms with additive label noise. In this work, we provide theoretical characterizations of the linear overparameterizee models in an MTL configuration for the first time. More specifically:
We provide explicit expressions describing the expected generalization error and knowledge transfer of multi-task Iearners and compare them to single-task learners in a lincar regrossion setup with i.i.d. Gaus- sian features and additive noise. Our results help to understand the system's 'ss behavior by highlighting the role and the interplay of various system parameters, including task similarity, model size, and sample size. For instance, we demonstrate that a peak exists in the generalization error curve of the multi-task learner when increasing model complexity, and the error peak stems not only from label noise but also from the dissimilarity across the tasks. We also highlight the effect of the sample size on the strength and location of the test error peak. Additionally, we measure knowledge transfer and indicate the conditions under which the tasks can be effectively learned together or interfere with cach other.
We provide similar results for continual learners and use them to explain the characteristies of replay- based CL methods. These results demonstrate the impact of replay buffer size on forgetting and the generalization error. Furthermore, we complete the theoretical results by analyzing the CL methods that use a combined strategy of relying on implicit regularization and rchearsal techniques. As a practical fruit, our results shed light on state-of-the-art replay-based CL methods by illustrating the effect of model size on reducing the effectiveness of memory buffer, especially in the presence of practical limitations where the replay buffer and model size can not be arbitrarily large.
We empirically show that our findings for the linear models are generalizable to DNNs by conducting various experiments using different architectures and practical datascts such as CIFAR-100, Imagenet-R, and CUB-200. Our experiments show that the test error follows a similar trend observed in linear models, indicating that our theoretical results can help understand MTL in DNNs. We also provide experiments with replay-based CL methods and study the effect of memory buffer size in practical setups with DNNs.
2 Related Work
Multi-Task Learning Several theoretical efforts have been made to understand the benefits of multi-task learning. In one branch, the statistical learning theory (SLT) toolkit was used to derive generalization bounds for MTL models. These efforts include using VC dimension (Baxter 2000), covering number (Pentina & Ben- Ben-David 2015), Rademacher complexity (Youscfi et al., (2018). and algorithmic stability (Zhang) 2015)). This line of work mostly focuses on defining notions of task similarity in the SLT sense (Ben-David & Borbely 2008) and deriving generalization bounds as function of the number of tasks, training set size, and task
2

---

Published in Transactions on Machine Learning Research (03/2025)
similarity (Zhang & Yang. 2022). However, these methods are inapplicable to the contemporary paradigm of overparametrized DNNs (Allen-Zhu et al. 2019), mainly because these models achieve zero training error and perfectly memorize the training data, yet they still generalize well to the unseen test set.
Additionally, some previous work studied multi-variate regression models, but their analysis or setup funda- mentally differs from ours. For instance, (Touunici et al. 2009) study sparse systems of equations with the goal of union support recovery (Kolar et al. 2011 Obozinski et al., 2011). More relevant to our work, Wul et al. (2220) analyzes two-layer architecture with shared module across tasks and separate output mod- ules for each task. They derive an upper bound on the performance of an optimal multi-task learning (MTL) model in a two-task scenario, assuming data abundance and finite model width, and examine the impact of task alignment on MTL performance. In contrast to our work, these studies do not characterize the exact convergence point of SGD in the MTL setup or explicitly derive the system's generalization performance in the overparameterized regime.
Continual Learning CL methods predominantly use either model regularization (Kirkpatrick el al. 2017) or experience replay techniques (Schaul et al. 2015 Rolnick et al.) 2019) to tackle catastrophic forgetting. Model regularization is based on consolidating the model parameters by penalizing drifts in the paramcter space during model updates with the goal of preserving the older knowledge (Kirkpatrick et al., 2017 Zenke
et al. 2017 Aljundi et al. 2018). The core idea for experience replay is to keep a portion of the training set a separate memory buffer (Schaul et al. 2015). These samples are representative of previously encountered distributions and are replayed back during learning subsequent tasks along with the current task data. The stored samples enable the model to revisit and learn from the distributions it has encountered before, facilitating the retention of acquired knowledge from past tasks (Rolnick et al.) 2019 Huang et al. 2024 Zhou et al. 2023). Memory-based methods have proven to be highly effective, and their variations often lead to the optimal approach in practice (Parisi et al. 2019).
Several theorctical papers exist on understanding continual learning models from different aspects (Doan jet al. 2021 Yin et al. 2021 Chen et al. 2022). The most relevant to our work examines linear CL models under a Gaussian data model when trained sequentially using SGD as the data for new tasks arrive. For instance, Goldfarb & Hand (2023) derives an upper bound for the generalization error of such models, and Lin et al. (2023) refines this result by proposing a closed-form characterization of the error. Furthermore, Goldfarb et al. (2024) studies the interplay between forgetting and task similarity as a function of different levels of overparametrization under a more general data model. More recently, Guha & Lakshman (2024) extends the thcorctical investigation to food-forward neural networks to show that increasing network widths to reduce forgeuting yields diminishing returns. Although these results are valuable and generally comply with our findings, they fall into the regularization CL-based methods. This is due to the fact that they merely rely on the implicit bias of SGD as regularizer that tends to the solutions closer to the weights of the previous task, while in our study, we equip the continual learner with an external memory buffer and study the closed form characteristics of the error as a function of different system parameters, especially the memory size. Moreover, some of the mentioned results (e.g.. Lin et al. 20233 can be seen as a special case of our studies in Section when memory size is zero.
Overparameterization and Double Descent In classical learning theory, it was widely accepted that increasing a model's complexity would initially reduce test error, but beyond a certain point, the error would rise again due to overfitting. However, the empirically observed double descent curve reveals that after the model complexity surpasses a certain threshold, making the model overparameterized, the test error surprisingly starts to docrcase again (Nakkiran et al., (021).. In fact, state-of-thh-art DNNs operate in the overparameterized regime, meaning that the model can perfectly fit the training data and achieve near-zero training errors while paradoxically still being able to generalize well to new unseen data, phenomenon known as benign overfitting (Zhang et al. 2017 Cao et al.
2022
For instance, linear regression exhibits similar behaviors under certain assumptions [Blkknn et al.] 2020). On the other hand, recent literature has pointed
Double descent is not specific to DNNs (Blkinn et al.
2019
out a direct connection between linear models and more complex models such as neural networks optimized
with SGD (Jacot et al. 2018 Gunasckar et al. 2018 Oymak & Soltanolkotabi] 2019). Consequently, several
studies investigated linear models as proxy for more complex models such DNNs (Hastie et al. 2022
3

---

Published in Transactions on Machine Learning Research (03/2025)
However, these works primarily focused on single-task setups and are not capable of capturing task notions in MTL setups. In contrast, our work analyzes the MTT, case where multiple tasks are learned together. Considering that even a simple multi-label classification problem is a special case of MTL, our work is essential in understanding commonly used overparameterized DNN models.
3
Theoretical Results on MTL in Overparameterized Regimes
We first define the problem setting and then offer our theoretical results.
3.1 Problem Formulation
Consider set of T learning tasks with linear ground truth model for each task. Specifically, for task t, assume input feature vector Tz Rs: and an output E R given by
y = n. 2i + 2;
(1)
where W, RR denoles the optimal parameters, and IR the random noise (Lin el al.) 2023 Belkin
let al., 2018). The specific true feature space for each task is unknown, and the features for a specific task may not be useful to other tasks. Since we scck to use single MTL model to learn all tasks, we consider larger parameter space with size p that encompasses all true features.
More formally, consider a global set of features indexed by 1,2, We assumne that the true set of features for task t is denoted by S, such that | = St- Among all, we choose a set of p features denoted by W to train our model, assuming that -T=1S C W. With this in mind, we define the expanded ground truth for task by introducing w, RR, where ut the same as my in the S, indices and filled by zero in the remaining p - 9, places (Lin et al.) 2023).
Data Model. We consider training dataset of size n, for task t represented as D, {( Vt. ) E where Fa,i E Rp and Ve, wf Zt,i. We also assume the features and noise are i.id. according to the following distributions:
Lt,i Ip) and 2e, ~10,00)
(2)
Here, o² denotes the noise strength.
Data Model in Matrix Form. Consider a matrix representation by slacking the training data as X, == [1 Tt,2, m. nt, Rpxm, Yt := [y,, Hr, st.., T and the noise vector as 2, [2r,1 2t,2, St,ne]
summarize the data generation process as
y1
X, w 22.
(3)
Similarly, we might stack the training data of all tasks to build X E RRxx, Vır E RR and 21.7 E IR where ? ñ = - ny the total number of training samples.
Single-Tee Task Learning (STL) To train a single-task learner for task t, we consider the standard setting in which the mean-squarod-orror loss function on D, optimized:
W; = arg min - IXX w - y/ 11.12.
(4)
WER*
Tht
In the underparameterized regime where p < nt, there unique solution minimizing the loss, given by i =X,X, In contrast, in the overparameterized regime where p >> ny, there are infinite solutions with zero training error. Among all solutions, we are particularly interested in the solution with minimum
4

---

To evaluate the generalization performance of task use Published in Transactions on Machine Learning Research (03/2025)
lg-norm, since it is the corresponding convergence point of applying stochastic gradient descent (SGD) Equation 4 Gunasekar et al. 2018) when starting from zero initialization. In fact, we in this case is obtained by equivalently solving the following constrained optimization:
W/: arg min | s.t.
X, w yr-
weRt
Since we are interested the overparameterizod regime, our focus on solving Equation
Single-Tar Task Generalization Error.
the following test loss:
Lr(w) Ex,=((x' -)² = ||0- wi1 +
(6)
In what follows, we drop o² and only use L((w) = ||we w 11 as a comparison criterion. As a prelude to our work, we review the following theoretical results on the generalization error of single-task learners:
Theorem 3.1. (Hastie et al. 2022)) When ne p 2 the single-task learner described in Equotion achieves
po²
E[Cr(we)]
The p- 1-1 and when p2 n4 + 2, the single-task learner described in Equation 3 oblains
[[[,(w))] = (1 14 11w
7₂0²
(8)
n4e
where the expectation is due to randomness of Xt and zt.
This result characterizes the generalization error of single-task learning (STL) across both the underparam- eterized and overparameterizeed regimes. To better understand this theorem, fix the number of training samples n and analyze how the error evolves as function of the number of parameters p. In the under- parameterized regime (p n) the learner doesn't have enough capacity to fit all training samples. This serves a regularization and leads to improved generalization, the learned parameters approximate the optimal solution with low bias but some variance. As increases in this regime, the variance grows, leading to an overall increase in error until the interpolation threshold (p 82 n) is reached. At this critical point, there exists exactly one predictor that can perfectly the training data. However, this solution is highly sensitive to noise, as small fluctuations the training data lead to large variations in the learned parameters, resulting in sharp spike in test error.
Beyond the interpolation threshold, p continues to increase, the model enters the overparameterized regime, where it can completely memorize the training data, driving the training error to zero. Despite this apparent overfitting, the model exhibits benign overfitting, where increasing, p paradoxically reduces the inffuence of noise ( n.o" yy-n,= + 0 p 20). As discussed in Section 2 this surprising behavior is associated with the implicit bias of SGD, which biases the optimization toward low-norm solutions. With this in mind,
we are ready to present our results multi-task learning.
3.2 Main Results on Multi Task Learning (MTL)
In this section, we present our main results for MTL and the deductions our results imply. Consider multi-task learner that simultaneously learns all tasks by solving the optimization given in Equation 5 using the training data of all tasks:
Wi.T arg min wll2 s.t. XT VIT.
(9)
WER*
To evaluate model performance, we utilize two metrics:

---

Published in Transactions on Machine Learning Research (03/2025)
Average generalization error reflects the overall generalization of the model, averaged across all lasks, i.e,
1 T
G(WIT) = T t=1
Σ (((IT))
(10)
Average knowledge transfer is metric to measure the gain of mutual cross-task knowledge transfer when comparing MTL to STL, ie.,
K(WIT) T Σ [(((s)) (.(w))).
(11)
Notably, lower generalization and higher knowledge transfer are more desirable. In what comes next, we offer the main results of the paper on the generalization error and knowledge transfer of an overparameterized multi-task learner.
Theorem 3.2. The multi-task learner described in Equation in the overparameterizad regime, where 22 2 has the following exact generalization error and knowledge transfer:
G[G(w)))
T
ne (1+ Tp
2(p ń 1|
Tn
w
11 +
n
1
no
(12)
term Ga
term G2
B[K(WIT) 2
Ry (1
Tn
Tp
2(p
term Ki
(13)
1
T
T4
2
T p The
n
(1+
T
n
where (wf, w denotes the inner product of the two vectors and the expectation is due to the randomness of X1.7 and 21:T-
Proofs for all theorems are provided in Appendix A To the best of our knowledge, this is the first theoretical analysis of multi-task learning in overparameeterize linear models providing an exact closed-form expression under non-ssymptotic conditions. Setting T recovers the prior STL results (Hastie et al. 2022). which means that our theorem is more generalized version of Theorem 3.1
Theorem 3.2 precisely characterizes various phenomena that occur in the multi-task setting, Terms G2 and G3, which also appear in the STL setup, correspond to the task norm and noise strength. However, term G1, which is specific to the MTL configuration, appears due to the distance between the optimal task vectors and is directly affected by task similarities. In fact, this observation motivates us to define a notion similarity between any two tasks i and as the Fuclidean distance between the optimal task vectors, i.e. [| - we l|2-
To achieve better insight into the results of Theorem 3.2 Figures la and 1b provide visualization of the empirical average generalization error for different values of p, σ, and task similarities, well as those predicted by our theory. Concretely, we have the following observations:
1. Interpolation threshold shifts. Similar to the single-task learner, an interpolation threshold exists for the multi-task learner. For both models, the average error increases at the threshold and decreases afterward. However, the interpolation threshold for the single-task learner occurs p = n4, while for the multi-task learner, it happens at p = n, which is the point where the number of MTL training samples matches the number of parameters.

---

Published in Transactions on Machine Learning Research (03/2025)
Aorio So Erow 60
sighterr arphiens P+C3
50 100
300 12:0
1000
53
1.0
000 1890 1000
120
500 1080
10000
100
500 1009
1000
(a) collaborative tasks
(b) conflicting tasks
(c) collaborative tasks
(d) conflicting tasks
The average generalization ion error of multi-task, single-task and continual learners w.r.t. to the model size p. (a) and (b) compare the MTL and STL for different levels of noise strength a. (c) and (d) present the generalization error of replay-based continual learners for different memory sizes m with zero noise. For all plots, T = 10, and for all tasks n4 = 50 and ||,]||² 1. In subfigures (a) and (c), the tasks are designed to be collaborative by adjusting (wi, ux) = cos g for every pair of task vectors. In subfigures (b) and (d), the tasks are chosen to be conflicting by setting (wt, the results from theoretical predictions. The dot marks the empirical evaluations averaged over 500 repetitions and are perfectly aligned with the theoretical results in the overparameterized regime.
Figure 1:
cos Za, The solid lines represent
2. Not all tasks can be lcarned together. The presence of the term ||w] w 112 in G1 indicates the impact of task similarity on the overall generalization performance. More specifically, if the lasks are highly similar, meaning that their optimal parameters are close, utilizing the training dataset of one task can improve the test generalization of the other task. However, when using a multi-task learner to simultaneously learn tasks with considerable gaps, interference happens in the parameters space and leads to poor performance.
The knowledge transfer also tightly depends on task similarity. In fact, the sign of the K1 term is closely correlated to the pairwise cosine similarity of the tasks. With conflicting tasks, negative knowledge transfer (Wang et al. 2019) happens and the multi-task learner underperforms the single-task learner. In addition to that, the K3 term in Equation 13 reveals that with the same p, the multi-task learner performs worse against the noise. However, to fairly compare the models, notice that the multi-task learner, which has only p paramoters, is being compared against an average of single-task learners with T p overall parameters.
Finally, collaboration or conflict across the tasks is observable in models with limited capacity. In other words, when p - oc, both G1 and Kj terms vanish, meaning that neither collaboration nor interference happens anymore in infinite wide models.
3. Task dissimilarity and noise intensify the error peak. Recall that the STL test error peak that exists al the interpolation threshold appears because of the high variance of the learned predictor. This behavior stems only from the label noise and intensifies as o grows larger. Nevertheless, when o = 0, mcaning that there exists no label noise the training sct, no error peak happens at the interpolation threshold of the single-task learner. Meanwhile, in an MTL setup, there are sources for the error peak. Not only the label noise cause the peak through the term G3, but task dissimilarity can also contribute through the term G1, meaning that highly dissimilar tasks can produce larger error peaks. In fact, the stochasticity of data points (i.e., X), together with high task dissimilarity, can lead to highly variant predictors with poor generalization behavior. This observation explains earlier empirical findings where the double-descent phenomenon was observed even in the absence of label noise (Nakkiran et al. 2021 and also distinguishes the study of MTL models from STL learners.
4. Descent floor exists when tasks are collaborative. When tasks are collaborative and the noise is not too strong, increasing in the overparameterized regime can be a double-edged sword. On one hand, increasing the number of parameters can help by reducing the effect of noise through the term G3. On the other hand, incrcasing p to co can kill the positive knowledge transfer across the tasks. There exists point in the middle where the effect of both mechanisms matches and a descent floor appears in the generalization error (see Figure Ta as an example).
7

---

Published in Transactions on Machine Learning Research (03/2025)
Although these observations are derived from our theoretical analysis of linear models, in Section 5 will use an empirical approach to demonstrate that DNNs trained with SGD also exhibit similar behaviors. Consequently, our theoretical insights can extend to and enhance the understanding of more complex models.
we
4 Theoretical Results on CL in Overparameterized Regimes
Continual learning can be considered a special type of MTL, with an extra constraint that the tasks arrive soquentially during continual episode. At each timestep t the model is only exposed to Dz, and the goal is to continually train a model that performs well across all tasks at the end of the episode. Note that the full data for past learned tasks is not accessible when the current task is being learned. In this case, sequentially fine-tuning on the most recent task at hand results in a model that performs well on the most recent task, but with degraded performance on previous tasks, a phenomenon which is called catastrophic forgetting. Formally, let w denote the weights of the continual learner at timestep t after sequentially observing tasks from 1 to t. Average forgetting (lower is better) reflects the amount of negative backward knowledge transfer in a continual learner, defined as F(uz))= == -ー Eiel [6((w>) Le(wr)]- For simplicity, in the rest of this section, we assume that n, = n for all tasks.
Implicit Regularization for Continual Learning. Prior to our work, recent theoretical CL studies relied on the implicit regularization of SGD to achieve continual learning algorithm and analyzed its charactcristics (Evron et al., 2022 Lin et al. 2023 Goldfarb & Hand 2023 Goldfarb et al., 2024). In other words, they start from zero initialization of weights, i.e. wa = O, and simply use SGD to lit the training data of the new tasks as the tasks arrive sequentially. Because of the implicit bias of SGD, this method is equivalent to the following optimization:
WFF1 = arg min - aw-wwl? s.t. X Ve+1,
(14)
WERP
where w = 0 E IRP. This continual learner, often called sequential fineluning in CL, is not elfective enough in practice and yields high forgetting. Even further, using explicit regularization can not solve the issue in practice and therefore, there is a need to study more useful CL strategies.
Replay-Based Continual Learning. Most successful methods in deep continual learning benefit from memory replay buffer (van de Ven et al. 2022). These methods store a small portion of the training data from each task. When learning new tasks, the stored samples are used to either regularize or retrain the model. In this section, we seek to theoretically study replay-based continual learning in the context of overparameterized lincar models. Specifically, assume we have access to m, samples for task in the memory bufler when solving lask L 1+1 1. Let the memory size be m, = Ei= m, We use the notation & E Rpxm, and d1:+ E R. to denote the training data and their labels in the memory.
A simple strategy to update the weights, in this case, is to ignore prior weights learned so far, start again from the zero-initialized weights, and use SGD to fit. both the data of the new lask well as the older dala stored in the memory buffer. The described replay-based contimual learning optimization is equivalent to
arg min |w|1 S.L. _ = yett, &T. = 91:-.
(15)
WER*
As opposed to Equation 14 the continual learner introduced here, tends toward low-norm solutions, but does not seck to minimize distance with the weights acquired so far from previous tasks. Therefore, we name this strategy pure replay-based continual learning.
Continual Learning. Lastly, we can think of a continual learner that uses mixed strategy of relying on both the replay memory buffer as well as the implicit regularization, ie., to store older samples in the memory and also sequentially update the weights using SGD without reinitializing to zero. This strategy can be formulated as solving
Implicit Reppariizattion-oop---------
W = arg min ||w - wrl2 s.t. X w = Btt1,
&T = P1.c.
(16)
WER*
8

---

Published in Transactions on Machine Learning Research (03/2025)
where wn = 0 RP. Notice that the optimization described in Equation 14 is special case of Equation 16 when = 0.
4.1
Theoretical Results on CL
In this section, we theoretically study the continual learners described in Equations 15 and T6 With closer look, solving Equation 15 at t = ' is a specific case of multi-task learning described in Equation achieved by setting n1 = mi...., -1 = mT-1 and = n. Therefore, we avoid repeating the theoretical results, and instead, we provide a corollary for the two-task case.
Theorem 4.1. Assume T = 2, 0 0, my m, n1 = n2 = n and ||= = || the continual learner described in Equation 15 it holds that
When 2 + m + 2, for
2m
+
B[G(w7)]= (1
+
Illo -usl-l
1|w 112.
(17)
2p
n
p (n + m)
p
term G₁
term G2
E[F(w7)] -(1 (1+
n
(n + m) ||w ww
m
2
m [ l
p
(18)
term F.
The generalization error of the regularization trepaay-bsedd learner is more complicated compared to the pure replay-based model and requires more delicate investigation. In fact, the dependence of wT to the samples in the memory buffer the source of such complication and produces many cross-terms in the final expression. To keep the results understandable and intuitive, we present the results for the two tasks case here and leave the full form for Appendix A.4
Theorem 4.2. Assume T 2, 0, mi the continual learner described in Equation 16 it holds that
m, ni = n22 = n and ||
|| When p 2 m + 2, for
[[G([=)]]
n2 (2-
m
2m
||ae
+
n
(1
n2 p
|| 11.
(19)
2p
m
p (n + m)
p m
term G1
term G2
B [[[(w)))]
- (1
n
m
- will )[wl
7
(1
71
(20)
p (n + m)
term
term
Equations 17 to 20 reveal the effect of memory capacity. Sctting m = 0 yields the results for the soquential learner described in Equation 14 Increasing the memory size reduces the forgetting and the generalization error through the terms G2 and F2, respectively. However, larger memory size can be harmful through the terms G and Fi if the tasks are highly dissimilar. Additionally, the full MTL model is recovered when m = n, meaning that all samples from the previous tasks are stored in the buffer.
An interesting observation the effect of model size on the forgetting and the generalization error. With larger p, both the positive and negative terms in the forgetting vanish. This observation suggests that bigger models with more capacity are less vulnerable to forgetting (Goldfarb & Hand 2023). Moreover, the next observation is that large p reduces the effect of memory size which means natural trade-off exists in practical applications with limited physical memory. In other words, one may consider investing the hardware in deploying larger models or consider a larger memory buffer for training samples. In fact, this is an essential result that sheds light on several state-of thetrrrttt--ttttte continual learning models where they reported that sample memory not enough and the best solution also keep some parts of the architocture the memory (Zhou al. 2023 Wang et al. 2022 Douillard al. 2022). We revisit this result in Section 5 when studying DNNs
9

---

Published in Transactions on Machine Learning Research (03/2025)
Although we presented the T 2 versions of the theorems for better comprehension, the mentioned phe- nomena are also observable in the T > 2 case as presented in Appendices A.3 and A.4 Figures [c and 1d visualize the error behavior of the replay-based model for different values of m when learning T = 10 tasks.
As demonstrated, the location of the error peak and its strength are affected by memory size, m. Morcover,
it surprisingly observable that the continual learner can somelimes outperform the MTL. baseline even when tasks are collaborative, especially at the multi-task learner's interpolation threshold.
5 Empirical Exploration for Deep Neural Networks
5.1 Empirical Results
Although fully understanding studying DNNs through theory scems infeasible, recent literature has pointed out connections between DNNs optimized by SGD and linear models and used these models a proxy to study DNNs (Please refer to Appendix[Boor discussion the connection between overparameterized DNNs and linear models). We follow a similar approach and empirically demonstrate how our results can provide insights for understanding and designing deep MTL and CL models. In sunmary, our most important observations are as follows:
MTL DNNs experience test error peak at the interpolation threshold, but unlike STL models, the peak not only depends the label noise but is also intensified when tasks are highly dissimilar ineffective paramcter sharing is happening (Figure 2), a consistent behavior across various practical datasets and architecture (Figures and
The distance of optimal task-specific parameters correlates with the amount of conflict that is happening across different tasks and can provide evidence for deciding which layers to decouple (Figure
Memory- based continual learners also undergo double descent as the number of parameters is increased. The characteristics of the error peak depend on the memory size as well as task-similarity and training size (Figure 6
High overparametrization can diminish the negative knowledge transfer (forgetting). Additionally, over- parametrized models benefit less from explicit memory buffers. Therefore, it important to adjust the resources accordingly between model parameters and raw training samples (Table 1
5.2 Experimental Setup
Our experiments are performed using three datasets and three backbone architectures to offer the general- izability and scalability of our results to non-linear models. It is notable that all experiments are repeated at least times, and the average values are reported. In the subsequent paragraphs, we present the most important details of our experiments and leave further information for Appendix
Datasets. We use CIFAR- 100, ImageNet-R, (IN-R), and CUB-200 datasets in our experiments. We gener- ate MTL or CL tasks by randomly splitting these datasets into 10 tasks, with an equal number of classes in each task (van de Von et al. 2022). No specific data augmentation or preprocossing techniques were applied, except for resizing to accommodate the input requirements of different backbones.
Architecture. We utilize the RosNet-18, ResNet-50 and ViT-B-16 as three different backbones our experiments. ResNet-50 and ViT-B-16 were respectively pretrained on Imagenet-1k and Imagenet-21k and used as frozen feature extractors shared across all tasks. However, used two versions of ResNet-18,, namely, frozen version, pretrained on CIFAR-10 and an unfrozen version with random initialization.
Our main focus is on the models operating in overparameterized regimes. Therefore, we leverage models that are complex enough to ensure that the training data is completely overfitted. When using the pretrained backbones, we implement a shared 3-layer MLP on top of the feature extractor followed by classification
head. We use the width of the MLP to manipulate the model's complexity by changing the number of hidden neurons for all of the 3 layers. On the other hand, when the backbone is trainable, we adjust the
10

---

Published in Transactions on Machine Learning Research (03/2025)
Tobotelh 0.3
03
28
15
320 0256
5352
0232
(a) Single-head
(b) Single-head
(c) Multi-head
(d) Multi-head
Figure 2: Comparing the performance of the single-task vs multi-task learners on CTFAR-100 using pretrained ResNet-18 backbone. The vertical axis is the cross-entropy loss and the horizontal axis is the width of the MLP on top of the backbone. a represents the noisy portion of the training set that was corrupted by randomly switching its labels. (a) and (b) correspond to the train and test loss with the single-head classifier, while (c) and (d) show the train and test loss of the multi- head architecture. Similar to linear models studied previously, deep MTL models also undergo the double descent behaviors with the test error peak depending the characteristics of the task similarity and strength of the noise.
model's complexity by considering different values for the number of convolutional filters in the backbone. In particular, the ResNet-18 has layers with [k, 2k, 4k; 8k] number of filters and we control the width of the backbone by changing the value of k. The typical ResNet-18 width has k = 64 filters. On top of this backbone, we use an MLP layer with a single hidden layer of size 128, followed by a classification head.
In all of the single-task experiments, consider two architectures for the MTL case. In the multi-head architecture, we employ a separate lincar classifier for each lask while in the single-head architecture, we use a shared linear head for all tasks.
use linear classification head as the final layer. However, we
Multi-task vs single-task lcarning. Figure presents the train and test behavior of multi-task DNN compared to single-task mod- els, when trained on CIFAR-100 using the pretrained 18.Net18. As depicted in Figures 2a and 2b train error is almost zero for heav- ily overparameterized models. However, the test error behavior is line with our theoretical analysis and is similar to the linear models illustrated Figure from several aspects: (i) The training interpola- tion threshold and the correspouding test loss peak for STL and MTL models are observable at different locations, which is consistent with the observation [ (ii) The error peak exists even with zero amount of label noise but is intensified with larger noise, which aligns with the observat ion 3 (iii) A descent floor exists in the overparameterrzed regime where the test loss is minimized, which is related to the obser- vation These obscrvations are crucial when designing MTL models and should be considered when optimizing the model size.
Average Inner Inner Product
0.0
-02
-0.4
layer
layer
-0.5
layer 3 classification head
16
256
4096
Figure 3: The average across-task inner product of task-specific opti- mal weights different layers for different values of MLP width, k. As this figure shows, the weights of the classification head has the most negative inner products which means that the tasks are highly con- flicting in this layer.
In Figures 2a and 2b while the single-head architecture is used and although the training error is zero for large enough models, we observe that single-task learners significantly outperform multi-task learners. Inspired by the observationss2 and 3 this behavior suggests that weight sharing is not happening effectively, and the tasks are not fully collab- orative, which can be considered a type of negative transfer learning. To dig deeper into this issue, we analyze the optimal weight for each task per MLP layer. In other words, start from fully shared model, repeatedly pick and unshare the weights of specific layer and optimize it to find the optimal weights of that layer per each task. Inspired by Equation
and
13 we calculate the inner product of the task-specific weights at each layer across different tasks and report the average values in Figure 3 For further details and discussion regarding this experiment, please refer to Appendix D.3
11

---

Published in Transactions on Machine Learning Research (03/2025)
3.0
35
2.5
1est L255
2.0
0.0
1.5
1.8
1.5
1.0
0.5
1.0
52
125
512
2043
5192
52
125
512
2048
8192
32
123
532
2045
6192
(a) CIPAR-100 / ResNet-50
(b) CUB ResNet-50
(c) IN-R / ResNet-50
1.75
2.5
3.0
2.5
2.00
1.5
2.0
1.0
1.5
0.50
0.5
1.0
0.00
20400 01.52
32
128
51.2
2048
8192
32
128
512
2048
8192
(d) CIFAR-100 VIT-B-16
(e) CUB / VIT-B-16
(I) IN-R VIT-B-16
Figure 5: Test loss of the single-task vs multi-task learners on different datasets with pretrained backbones. The vertical axis is the cross-entropy loss and the horizontal axis is the width of the MLP on top of the backbone. a represents the noisy portion of the training set that was corrupted by randomly switching its labels. The multi-head architecture was used in all MTT experiments. These plots highlight the difference between MTL and STL double descent across several practical backbones and datasets.
According to Figure we observe that the weights at the final clas- sification layer have the most negative inner product values. This observation suggests that CLEAR-100 classes are highly conflicting the classification layer. Nevertheless, extending the classification head the multi-head architecture enables effective knowledge transfer in addition to improved performance for the multi-task learner, il- lustrated in Figure 2d In other words, while unsharing any layer benefit the MTL performance (as elaborated in Appendix D.3). un- sharing the classification head is the most effective approach, which Icads to desired behavior closcly similar to the collaborative ver- sion of the linear models Figure Ta As an additional experiment, we present Figure 4 which provides a similar comparison using an unfrozen version of ResNet-18 training from random initialization. Al- though this backbone is much deeper compared to the linear models we studied in Section 3.2 the consistent behavior suggests that our findings also generalize to more complex architectures.
10
100
Textose
Figure 4: Test loss of the single- task (Top) multi-task (Bottom) learners on CIFAR-100 with un- frozen ResNet-18, The vertical axis is the cross-entropy loss and the horizontal axis the scale factor that controls the number of filters convolutional layers. As we observe here the test loss behaves similarly to the case where only the MLP head is trained.
Additionally, Figure offers further experiments on CUB and IN-R datasets trained on two different pretrained backbones. Observing similar phenomena what we presented in observations 13 endorses our hypothesis regarding the connection between results for linear models and DNNs in MTL setup. For example, in all of these experiments the test error peak of the multi-task learner is considerably strong, while in some configurations, the STL error peak is negligible. This observation emphasizes that the label noise is not the sole source of the error peak; rathor, as predicted from our theoretical analysis, the inherent dissimilarity across different tasks will also contribute the strength of the peak. We conclude that results help explain
12

---

Published in Transactions on Machine Learning Research (03/2025)
Tes LOSS
255
10384
(a)
Test Loss
(b) Forgetting
(c) Test Loss
(d) Forgetting
Figure 6: The Lest. loss and forgetting of continual learner on CTFAR-100 w.r.t. width of the model, k, for different levels of per-class memory capacity. m 0 means naive fine-tuning, and 500 corresponds to the full MTL sctup. (a) and (b) correspond to the experiments with the Pretrained ResNet-11 and subfigures (c) and (d) show the results when fine-tuning a ResNet-18 model from scratch. The multi-head archilecture with no label noise was used these experiments. These figures show that a CL learner also has a test error peak which is affected by the size the memory buffer.
the behavior of DNNs, and combined with empirical investigation, they can be used for optimal design. Complementary experiments, as well as the full training and testing curves of MTL experiments, are provided in Appendix D.1
Replay-based continual learning. Figure G presents the test loss and the forgetting of a DNN continual learner on CIFAR-100 for different amounts of memory budget and model size. These models follow trend very similar to the linear models presented in Section ! In particular, there exists a test eгror peak for all modols, but its location and strength are controlled by the memory size. By incrcasing the memory size, the test error peak naturally shifts to the right with reduced intensity. Another observation aligning with the linear models is that the forgetting greatly reduces for large model sizes, even without any memory, a phenomenon also confirmed by recent studies on sequential fine-tuning CL (Goldfarb & Hand 2023).
Additionally, it is noticeable that increasing the memory capacity does not always uniformly increase the performance for all values of model size. Especially, the memory size is less effective with larger models. This raises fundamental question when designing CL models in practice. Given a fixed budget for physical memory, is more efficient to spend on increasing the model size or building a bigger sample memory? Table T contains the results of experiments with varying budgets dedicated to the replay memory buffer. As the results show, different optimal balance points exist for all datasots that maximize the avcrage accuracy and minimize forgetting. This observation aligus with the recent CL methods, where an equal portion of the sample memory budget is devoted to specialized blocks (Zhou et al. 2023 Wang et al. (022)) task-specific tokens (Douillard et al. 2022). Therefore, we propose that our theoretical findings in lincar models can be applied to practical CL algorithms to gain a deeper understanding.
The effect of varying sample memory budget and architecture memory budget over the average accuracy and average forgetting of overparameterized continual learners on two datasets. All values in this table are presented as percentages. The sample budget lincarly proportional to the number training images stored in the memory, where 100% corresponds to storing 10% of the whole training set. A complete version of this table with more details presented in Appendix
Table 1:
CIFAR-100
Imagenei-R
CUB
PMNIST
Mem. Budg.
Arch. Budg.
Avg.
Avg.
Avg.
Avg.
Avg.
Avg.
Avg.
Avg.
Acc.
Forg.
Acc.
Forg,
Acc.
Forg.
Acc.
Forg.
100
<1
56.6204
-22.510.5
76.1204
-3.300
87 87.620.2
-4.520.2
32.812.8
-69.823.0
80
20
70.6-u.1
-7.140.1
76.94a.
-2.0+u.3
84.140.5
-6.7
82.941.
-13.9+1.1
60
40
71.9+0.2
-5.4+0.3
77.140.6
-1.4+0.4
88.5+0.
-1.8+0.8
87.740.4
-8.5+0.4
60
66.3_0.4
-11.3+0.4
78.9
-1.0+n.1
92.0+0.1
-0.110.2
88.8
-7.120.1
20
80
67.4 67.4_0.3
-9.7+0.3
78.520.3
-0.640.1
91.7+0.4
-0.45n.3
88,240.3
-7.2-0.2
100
63.8_o.3
-10.810.3
49.520.6
-20.5
75.810.7
-12.210.7
68.2+1.
-17.910.2
13

---

Published in Transactions on Machine Learning Research (03/2025)
6 Conclusion
We studied overparameterized lincar models in the MTL and CL settings and derived exact theorctical char- acterizations to better understand the impact of various system parameters over the average generalization error and knowledge transfer. We also analyzed replay-based continual learning and provided theoretical results to describe the generalization error and forgetting of such methods. We also performed extensive experiments with DNNs to show that similar behaviors are observable the deep MTL and CL models, and understanding the linear overparameterized models can guide our intuition when studying more complex models. Please see Appendix E for discussion the limitations and future work.
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 134, 2018.
Zeyuan Allen- Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advonces in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Jonathan Baxter. A model of inductive bias learning. J. Artif. Int. Res., 12(1):149-198, March 2000.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 541 149. PMLR, 10 15 Jul 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-133 15554, 2019.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal Mathematicss of Data Science, 2(4):1167-1180, 2020.
Shai Ben-David and Reba Schuller Borbely. A notion of task relatedness yielding provable multiple-task learning guarantees. Machine Learning, (3(3):2782287 Dec 2008.
Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. In Advances in Newral Information Processing Systems, volume 35, pp. 25237-25250. Curran Associates, Inc., 2022.
Rich Caruana. Multitask learning. Machine Learning, 28(1):41 75, Jul 1997.
Chen, Christos Papadimitrion, and Binghui Peng- Memory bounds for continual learning. In 2022 IEEE 63rd
Arauul Symposium on Poundations of Computer Science (FOCS), pp. 519 530, 2022.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming, 2020.
Michael Crawshaw. Multi-Lask learuing with deep neural networks: A survey. Arxiv, abs/2009.09796 2020.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE Signal Processing Magazine, 29(6):141 142, 2012.
Thang Doan, Mehdi Abbana Benmani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 1072-1080. PMLR, 13-15 Apr 2021.
Arthur Douillard, Alexandre Ramé, Guillaume Conairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Ity Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How calastrophic can calastrophic forgetting be in linear regression? In Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp. 4028 4079. PMIR, 02 05 Jul 2022.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Robert M French. Catastrophic lorgelting in connectionist networks. Trends in cogroiloee sciences, 3(4):128 135, 1999.
Daniel Goldfarb and Paul Hand. Analysis of catastrophic forgetting for random orthogonal transformation tasks in the overparameterized regime. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 2975-2993 2975-9993. PMLR, 25-27 Apr 2023.
Daniel Goldfarb, Iay Evron, Nir Weinberger, Daniel Soudry, and PAul HAnd. The joint effect of task similarity and overparameterization on calastrophic lorgetting analytical model. In The Twelfth International Conference on Learning Representations, 2024.
Etash Kumar Guha and Vihan Lakshman. On the diminishing returns of width for continual learning. In Forty-first International Conference on Machine Learning, 2024.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1832-1841. PMLR, 10-15 Jul 2018.
Trevor Hastie, Andrea Montanari, Saharon Rossel, and Ryan Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of
generalization. ICCV, 2021.
Zhehao Huang, Tao Li, Chenhe Yuan, Yingwen Wu, and Xiaolin Huang. Online continual learning via logit adjusted softmax. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advonces in neurol information processing systems, 31, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran et al. Overcoming catastrophic forgetting in
Milan, John Quan, Tiago Ramalho, Agnieszka neural networks. Proceedings of the national academy of sciences, 114(13):3521-333 2222266 2017.
Mladen Kolar, John Lafferty, and Larry Wasserman. Union support recovery in multi-task learning. Journal of Muchine Learning Research, 12(72):2115 2435, 2011.
Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers of features from tiny images. 2009.
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. Proceedings of the 40th International Conference on Machine Learning, (CML'23. JMLR.org, 2023.
Karim Lounici, Massimiliano Pontil, Alexandre B. Tsybakov, and Sara van de Geer. Taking advantage of sparsity in multi-task learning, 2009.
Preetum Nakkiran, Gal Kaplun, Yaminí Bansal, Tristan Yang, Boaz Barak, and Tlya Sutskever. Deep double descent: where bigger models and more data hurt*, Journal of Statistical Mechanics: Theory and Experiment, 2021 124003, dec 2021.
Guillaume Obozinski, Martin J. Wainwright, and Michael I. Jordan. Support union recovery in high-dimensional multivariate regression. The Annals of Statistics, 39(1):1-47, 2011.
Samet Oymak and Mahdi Soltanolkotabi. Overparameterized noulinear learning: Gradient descent takes the shortest path? In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
PMLR, 09-15 Jun 2019.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 71, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Autiga, et al. Pytorch: An imperative style, high-performance deep learning library. A ddvances neural information processing systems, 32, 2019.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Anastasia Pentina and Shai Ben-David. Multi-lask and lifelong learning of kernels. In Algoritlonic Learning Theury, pp. 194-208, Cham, 2015. Springer International Publishing.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Tom Schaul, John Quan, Joannis Antonoglou, and David Silver. arXim:1511.05522 2015.
Prioritized experience replay. arXiv preprint
Gido M. van de Ven, Tinee Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185 1197, Dec 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset, 2011. California Institute of Technology.
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. arXiv preprint Xix::2000 04662, 2022.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVE conference computer vision and pallern recograltion, pp. 11293 11302, 2019.
Sen Wu, Hongyang R. Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference Learning Regresentations, 2020.
Dong Yin, Mehrdad Farajtabar, Ang Li, Nir Levine, and Alex Mott. Optimization and generalization of regularization- based continual learning: loss approximation viewpoint, 2021.
Niloofar Yousefi, Yunwen Lei, Marius Kloft, Mansooreh Mollaghasemi, and Georgios C. Anagnostopoulos. Local rademacher bomplexity-based learning guarantees for multi-task learning. Journal of Machine Learning Research, 19(38):1-47, 2018
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference Learning Represenlations, 2017.
Yu Zhang. Multi-task learning and algorithmic stability. Proceedings of the AAAI Conference on Artificial Inteltigence, 29(1), Feb. 2015.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586 5609, 2022.
Da-Wa Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan, A model or 603 exemplars: Towards memooy--fficien class-ines inseemenrllttlll learning. In International Conference on Learning Representations, 2023.
16

---

Published in Transactions on Machine Learning Research (03/2025)
A Proof of Theorems.
A.1 Useful Lemmas
In this section, we start by providing some useful lemmas and then provide proofs for the theorems in the main text.
Lemma A.1. Consider square invertible matria S E R"xa, Assume S can be partitioned into four smaller blocks as
A C D
B
S=
where and are invertible square matrices with arbitrary relotive sizes. Denote D CA-1B and assume that H is invertible. Then, the inverse of S can be written
-1 A-1 A- BH
-A-1BH-1
S = C D =
CAA
H-1
Proof. The lemma can be proved by simply examining that S8 1.
Lemma A.2. Consider square invertible matriz S Ruxn
Assume S can be partitioned into 3 3x2 3 blocks:
E
F
G
S=
H
J
K
L
M
N
Then the inverse of S is
1 + --
-F-1FUUR-Q)A4-
-E--UR- BR-
S-
(H-B
-R-IVE-
-R-1CA-1
R-1
where we define
A= J - F
= K HE
C= M LE-IF
D N LE-1G
U = G FA
V = L CA-
R= D-
and we assume. that the matrix inverses erist wherever necessory.
Proof. Similarly this lemma proved by examining S5-1
17

---

Published in Transactions on Machine Learning Research (03/2025)
Notations. We have already introduced some matrix notations in Section In this section, continue to introduce new helpful notations. Remember that for task E (7) with samples, we had X E Rex", " E RP, E R and yr- = =X,"
z
Now consider indices i,j E [T] such that i j and denote nij = Ei= nr. We introduce the concatenated data matrix as Xij E g2xms which is constructed by concatenating data matrices from X, to X2 in the Xy] With this notation, the matrix representation of all training samples is XIT E _ where n E ft. Additionally, we introduce the matrix Xuje Rxxa which is built from X1.T by replacing all entries with zeros except at the columns corresponding to the task L. In other words, Xop= [0pxr Xx Opxa], where Opxr is an all zero matrix with size p x r, E "4
second dimension, i.c. Xi = [X; Xi+1
and g 9= Ei=41 nz.
Since our study mainly focuses on overparameterized regimes, the data points X₁ E R are random matrices with more rows than columns. Therefore, (x, Xx) exists almost surely and we define X (X, x,) and the projection matrix P. XX, X,) 'X,"" Additionally, let P.j = Xx,xX
x X1y (x] and Poj- = (XT Xooe With this in mind, we provide some useful
lemmas
Lemma A.3. Let X1.2 E (px(() the resull of concalenating X1 E Roxa, and X2 E Rpxa2, Also let Xoj1 E ( be result of concatenating X1 with zero matriz. Similarly, build Xoj2 E Rex(my by concatenating a zero matric with X2. Assuming p > ni n2 and that all inverses cxist, it holds that
i. P1.2 (12
P - P1)
7. Poj1 X1,2(X (112
1122 0f1 Pi - - -
ii. Popp X112 1:2 112)
Xop (I PX) -.)
iv. Pol1 = Xo11 X12 = P PX (1 P
x12 = X1:2(XT2X113
6.
= =[x|- -x--(1- (T- 22) (T P))x)) --) --
where Pi = -X x = X1 and I is the identily matrix with size p X p.
Proof. We start by rewriting X12 block matrix and then finding its inverse:
X
xT -1
(x12
X2 X1 X2 X2
Now using Lemma
A.1
[,,,,))"10000,""",,,3>>3"<"X"""" -1=
-(X -
-H -H-IX X,,xXXxxX X,)
H
where H H= =XXXX, XX, x
Now, by noticing that X1:2 [X1 Xz].
we can prove what was desired by doing multiplications:
Notice that Pops by definition projection matrix and we use this notation just for consistency.
18

---

Published in Transactions on Machine Learning Research (03/2025)
i.
([x]x)
xTx]
X
P122 = [X1 X2]
X2
= P1 P1 - + XXH
= (1- XX
ii.
(x/)1) x )
(xx x[x]
x
=01==[Xx [X1 X2]
[0]
= = + P
P.H.H x Pi X2
P1 - (I - (x -
iff.
(xxxx) x
-1
pop= = [X1 Xa]
[x X2 X2
= - - X + x2 X
= -T -I (x (1 -1-
iv.
( X1
xx
[
PJT Pop1
[X1 0|
Xı
P. = P1 = =P -
--)
(, -,
X2H-1
x/22 = [X] X3]
-I-*X]XXXXXXX
111
=[x| - (I -I-- -P)))x(XX (-X ((-R)x(X((1PP)))))
Lemma A.4. Let XLa E (ex(m)nntts)) be the result of concalenating X1 E Rpxxi, , X2 E RRx=2 and X3 E RPx=s, Also let Xn2, E (") be the result of concatenating X1, X2 and X3 with zero matrices such that Xoj1 = [X 0
0 Xape = lo X2 0] and Xuj3 = [0 0 Xz). Define Pon = X1a(XTT (aa(x -1x) on1 and Pojs X ( Xo3 Assuming p > ny + n2 + na and that all inverses erist, it holds that
P PPP
where Pi = Xx)(XXXx1) and denote Xx (1- -1) (P-)).
19

---

Published in Transactions on Machine Learning Research (03/2025)
Proof.
Poj1 Po3 Xo/1(X X13)) - X1:3 (1:3)) XX
= (1 -xx3 Xjj3
X1
x X2
X X3
-1
X
X2 X1
X2 X2
X2 Xa
X
Xj X1
Xg X2
Xg X3
Now Lemma A.2 to find the matrix inverse. With the notation introduced in that lemma, looking for -X X3. Thus have,
A=J-ME
B= B-K-HE'G=X
C =
D= N I- ( 81)x3
U=G-FA B X P122)33
D= B X - PP)(( P1:2
PT Pol3 = -,, UR
- x = - -1- -120- -1a)) -x2
Lemma A.5. Assume X E Rexn is the training dala malrir and E RD are the corresponding labels. In the underparameterizee regime where < n, we seek to solve an optimization of the form
w arg min X -vll.
WEBP
The optimal solution to this optimization is given by
w' = (XX (XXT)-Ixx
Proof. By setting the derivative of the objective to zero, we obtain:
X(XT w- - -y) = 0 - (XX (XX
noticing that (XXT)-1 exist almost surely if X Gaussian random matrix with p n.
Lemma A.6. Let X Roxn be the training data matria and y E RD be the corresponding labels. Assume
Wo De is on arbitrary fized vector. In the overparameterized regime where p n, we seek solve oplimizations of the form
20

---

Published in Transactions on Machine Learning Research (03/2025)
w" = arg min ||w - woll2 8. xTw w
We
The optimal solution to this optimization is given by
w" (I- P)wo x
where = X(XXx)-IX and x
Proof. Using the Lagrange multipliers and by setting the derivatives to 0, we can get:
1
w", x arg min A,
7 511u mol1 x (X w y).
= w wi + -Xx 0 = "² XXX + wi
X w0 - XX + x
wo X (X x x wo (X x)-
nn = -x(xxx) x w/ +X(X xx=xx)_1 y + wp = (I P)ws + Xty
21

---

Published in Transactions on Machine Learning Research (03/2025)
Lemma A.7. Assume matrices X1 E Rpxm and X2 IR to be random matrices with entries being independently sampled from the slandard Gaussian distritulion ((0,1), Assume p + n2 + and let P. X1) [ be the orthogonal projection matria that projects onto the column space of X1. Also assume X1:2 E (ex((1+a2)) be the result of concatenation of X1 and X2, and Xo11 E (px be the result of concatenation of X1 with a zero matrir. Additionally, let P1.2 X12(X122 X1.2 and Poj
XX]1- Assuming w E Ip' is a fized given vector, the following equalities hold:
4. GIlP 2i ||w|2
a. Biil!/ - Pj P1)x01l]]] = (1- - 1y |w|l2
M B = 22 (1 p- (z) + - |0|
p
Proof.
i. Without loss of generality, we can focus on finding Pu| where a e Rº and || 1, since X1 is Gaussian matrix with rank n1, and Pi = (x, (X) Xx is an orthogonal projection matrix with a similar rank. Due to the rotational invariance of the standard normal distribution, we can assume that P1 a fixed matrix and instead, u is a random vector uniformly sampled from the unit sphere in Iº. Using the rotational invariance again, we may assume without the loss of generality that P1 the coordinate projection onto the first n2 coordinates in RP. Then it holds that:
Ex, [| = E.,
71
P
ii I- Pi is a projection orthogonal to Pt. Therefore, by Pythagorean theorem,
E Ei Pj [)] |w:1/ E|] P1w| (1
n1 |w:l]
iii.
E|| Pol E(w Poj Poj
E(w Piw] + F(wJ (T -1)x2)-111 Pww]
(Lemma A.3 part (iv))
n1 | + E[tr(w] PxXx(X,) (1-)))
21) 'x, P1w)]
| + Ex, (tr,,te(xx, (X] Pawe T PXXx (1 -1-))) p
To find the above expectation over X2, notice that P1 is an orthogonal projection matrix and, therefore, for fixed P1, P, X2 is independent of (T X2)
P
w/
n1 Ex, [tr(Ex, www (X
|| p
n1 p
||ll| + Ex, [tr([t(( IX), (I (1-
||ll|2 + Ex, 12 -
p
22

---

Published in Transactions on Machine Learning Research (03/2025)
We focus finding the inner expectation first. Notice that for fixed X1, I-P; projection matrix with rank p- Due to the rotational invariance of the standard normal distribution, we may assume without loss of generality that 1 - P1 is the projection matrix that projects onto the first p - n1 coordinates. With this in mind, (X2 (I - 22) -1 follows an inverse- Wishart distribution with an identity scale matrix Ingxna E 22 and p - 71 degrees of freedom.
Pi is an orthogonal
Po
w||i
n1
||ol
+ Ex,
P1wll tr(
I X
)]
n1 12
||=11 2 +
V22
Ex, 11111
p
p
(n1 n22) -
11
(1 + p- (πi + 122) - |||11|
112
Lemma A.8. Assume matrices X1 E RRxx, X2 Rpxna and X3. E Rpxxa to be rundom matri- ces with entries being independently sampled from the standard Gaussian distribution (0,1)). Assume p n1 + n2 + na 1 and let X1.3 ppxtmntnatts) be the result of concatenation of X1, X2, X3. Also let
Xo11 x, Xo E RPx((m +n=+na) be the result of concatenating X1, X2 and X3 with zero matrices such that
Xo11 [X] 0 0] X [0 X2 0] and X013 = [0 0 X3). Define Poj = X13 (1:3))133) XX and
Poj3 XI: (X TT x1.)) XT Assuming w, w E TR? are fived given veclors, il holds that.
w Poji Pojzw'|
113
(w, w')
p(p - (n1 + ne + n2s) -1
Proof. Based on Lemma
A.4 we have
E/w T Pop Pops'' -E(w PI(
P - P122) - -12)))) X3ux w']
-xx, Exg iw PA(I (X,) (I- P.) Xx)) w'll
where P1:2 = - (T- Now denote p= (I P.) P..2). With fixed X1 and X2, P is orthogonal projection matrix, meaning that PT p2 P. Therefore,
Pol Pojaw'] = Xx,,,, Ex, (w T Pi(T P12) (X, (P+ P)wo[]] PXs) - (1x Xg] Pw']]
-Ex,, Ex (w Pi(I
-xx,,x, (w T -111 PXx) PI
Notice that for a fixed X, and X2, Pl and P are two orthogonal projection matrices. Therefore, the following independence relations hold:
- PI 11 PX2
Pi(I P12) 11 Pi)(I P1.)Xs
With this in mind, we write
23

---

Published in Transactions on Machine Learning Research (03/2025)
Elw' T Poj1 Po1sw`| = -Ex,,x, ( Pxxx II P1.2) (X3 xx PX
- Xx [t( (1 Pju'vi w T Pi( - (xx)) (Xx)
= =0 tr [x3 -I- P)w'u w P1 P1. X3 Ex PXx) PXx
-Ex,,x,(tr((T P)w' Pi(T Pı.2) tr(Ex, (Xg PX3)=111
Here again technique similar to the one used in Lemma A.7 part (iii) by first focusing the inner expectation. Assume a fixed X1 and X2. Then I- P1 and PI are orthogonal projections with ranks ny and #1 respectively. Additionally, (T - -P)) 22 is an orthogonal projection matrix with rank n2 and p- n1 > ng. On the other hand, P1 and (1- P12 are orthogonal projections. Therefore, P1 is a projection matrix with rank n, + ng and P I- (PI + P1) projection matrix with rank p- (m1+m2) When taking the inner expectation only w.r.t. X3, due the rotational invariance property, we may assume that (T PT) Pi:z) the coordinate projection onto the first p (n1 n2) coordinates. Therefore, (X3 (I PI)(1 Xz) IWV(I P (n1 n2)) and we have:
+ -Ew T Poji Popsw1| = -x,, [tr((1 - P)w' T TI(I P1:2)) ((2)) tr(
I Xna
})
p (m1 + 12) - -13
113
E Ex,xx_(tr(w P PI ((2))1.2)11- -1ao'((-PP)'')
(m) + n2 + n4)
Now, using the definition of and P1.2, we have
E(w Poj Po:3w1]
713
Elir(w] PP( P12))u))
p - (n1 n2 + nz)
113
E(tr(w' P(w')l
p (ni n2 + 113)
+ (m1 + 72 + n2) - E(tr(w Px PA)X)) (I- P:)w')l
n1
Remember that for given X1, P1X2 and - P2)X2 are independent. Therefore,
E|w Pojz Polsw']
E[tr(w] ((iT()) P(w')]
=
p (ni + n12 na))
n1
x
1
Bl(w w + wa' Pi(w + w') -w-w') Pi w')l
p- (ni + n2 + n23) -
4
113.
x
1
[||l (w w w2||2 |P (wo w')l
(ni + 122 + n13) -
72
1
(|| w 1/2-1w 1/wwww113 w0(l3)
(Lemma A.7 part (i))
p (ni + 12 Tt3)
4
p(p (ni + 11
(w, w²)
+ n23) - 1)
24

---

Published in Transactions on Machine Learning Research (03/2025)
As concluding remark to this section, notice that When X is Gaussian matrix with iid entries, X is almost surely invertible (i.e. the probability measure of x X not being invertible is zero). Therefore, in what comes next, we assume that all Gaussian matrices in this form are invertible without loss of generality.
A.2 Proof of Theorems
Now that we have all the required building blocks, we are ready to provide the proofs. We start by proving the theorem related to the single-task learner and continue by proving the theorems on multi-task learning and continual learning.
A.2.1 Proof of Theorem 3.1
Proof. underparameterizeed regime. In the underparameterrized regime where ne 2 p+ 2, the signle-task learner the unique solution to the Equation and is obtained by wt = (X,X,))-XXX according to Lemma Therefore, we must. have
(X.X,) XeM. = -Xxx) Xx(, + = w/ + (X,X,)) X
[w)] Billw; w² E(I((XX;)) Xx,21
where the expectation is due to randomness of both X, and 2. First, we take the expectation w.r.t. 24 Nn. (0, 0² );
B[G((w)] = Bi= X (xxx) X
[x]
=0xE]tr((xxxx]] xx,x,
==xx](r((Xxxx]) )
= o² (x,,, ID)
I Txxp
r2 tr(
111 - p
2o²
74 p 1
where the two last. line comes from the fact that (X2X7)-1 follows the Inverse-Wishart distribution as [W(ppxp n).
overparameterized regime. In the overparameterized regime where p 2 ne + 2, we look for the solution of optimization 5 Therefore, based on Lemma A.6 have
25

---

Published in Transactions on Machine Learning Research (03/2025)
[w)] Blllw; wi [1²]
-BIlx//v - will
=XIIXX wi + zx) - - [|
=EIl([ Po)uy - xx [|2]
EIIl(I Pt) [12] + B[|| X z |1 200 2 B P.)x
Elll(I + Bi2 (x, X1)
[E[2d) =0)
noo²
E|I((T Pf) 1|2]
(x X.) ~ IV xn;
p n4 -1
(1 T4 |10 + 71,02 2
(Lemma
A.7 part (ii)
26

---

Published in Transactions on Machine Learning Research (03/2025)
A.2.2 Proof of Theorem 3.2
Proof. Based on Lemma A.6 we can write
wi. X1xr (X X1:) (1:)) V1.r
Using the notations introduced in the previous section, we have
W (X E ", + 21:T) = Σ Po + x1
Now we start by calculating the multi-task learner's loss for the task i:
E [[C,(WIT)] = BI] -
T
= =1l
Σ
2s + +1:1 - w/ 11
(w) - w/) + (1 - P(rT) + 2
=
T
=
|II|
Pojs (w -w?) + (1 PI.T P.r (u B 11
(A1T 11 XI.T B[zrr] = 0)
T
[1]
Poje (w, - w()) + (1 Purr) (1((( (20,~~(00021))
2o²
E[|
(w w:) + (I P(rr)
((XITX17)-' IW(Jax..P))
7
Therefore, we focus on finding the first term by expanding it. First notice that
U-PPi Pol) = - - -xT
XXx -XT. ( -1oT=
Thus,
T Σ
Poj (w; w³) + (1 - PPr))w?11
T
EII
(w - - - P1r))=i111
T
[||
Poja(w - - + (1
(Lemma A.7 part (ii))
77
Ell| -w])ll +
El(w; wt) POT Pow ww]] + (1
5 5'
27

---

Published in Transactions on Machine Learning Research (03/2025)
Since the distribution of Pojs is invariant to the permutation of columns of X1.7, we focus finding Poj (wi - ][ and [B((w] wF) T Porr - wf)] without loss of generality. In Lemmas A.7 and have already calculated similar quantitics. Therefore,
A.8
Pol w
n1 (1
n1
||wof w 11
72
and,
E[(w] - w P
Porr(wg - w]]]
p(p - nt - 1) (w0 w/, w - wx))
111-111
Overall, we can write:
[[Cx(wiT)
T
n Rgy
(w5
wT, w w;) + Σ
n2g (1 +
71 R2
}1 [ w ||
1
p
s=1
8
+ (1
71
>|w
|12
nio²
p
p ñ
T
T
T
Rg
n2 (1 +
7
(w5
w7, w
w.) +
m/w - w
p
#=1
Ti02
+
(1
||0
11
+
p ñ.
T
T
n ng
(110g 10..112 - - w 112 - 1 - 11
2p
8=1
T
ns
+
7
)| w|'1 + (1
71 p
||wl 1| +
P
p
p Fr
T
||:
w 11
ng 1
w 112 + (1
72
no2
(21)
2p
p
p
p 7 1
Now we calculate the desired metrics as
T
[[G(wir)) = T i=1
(wir))
T
T
T
1
|| - w/ 1|?
+
[w
-
w1
2p
P
T
1=1
T Σ
n
||
1|2 +
p
p 1
1=1
T
T
(1 +
n4:
)|
-w,'l
+
1
(1
| l|
T
p
p
T
p
p
1
28

---

Published in Transactions on Machine Learning Research (03/2025)
and
T
E [[K(WIT)]
1
Σ
B B[C,(w:)] - [[C((wrT]]
= T
T
T +
n1,0² 2
Σ
T
T
f1 -
i=1
T
T
T
n
(1
+
-wyll
w
1
(1
71
|
no?
T
p
12
T
p
1
1=1
8=1
==1
1 T n1:02
|||
+
Σ
p
T
p 71
i=1
nh p
(1 +
p n2 /1|25]1 + 1|w,511 2(2= w,))
n4
1 T
| 11
p 72 1
2
ng
(1
T 2
Yw w,}
T
n4
(T
T n
)|w,'l
p 7
p
p 71
T 2 n4
T
ña 2
)|
| ||
p- n
p
p - n
T
Σ 1=1
n4
|| 112
+
n2o²
p
T
114
T
T
2
2 nz
(1
Yw w
T
7
T
Tng
ng
2 7
7
2
n1
ng
||
T
P
p p
n1
1
pp-
s=1
T
RGO² 2
ñO²
T
p
T4
1
p
2
T
(1
T
2 nzs
1w 3, w
71 (1
1
+
n2
|w 1
p
p
p
T
p
7
i=1
4=1
s=1 l
1
T
R2g02
2
Rd²
T
p
74
p 7
29

---

Published in Transactions on Machine Learning Research (03/2025)
A.3 Proof of Theorem 4.1
Proof. To provide the proof, we start by finding the loss of task at timestep 4, i.e. (w(;(())
}} using an
intermediate result in the proof provided in section A.2.2 In fact, we refer to Equation 21 and assume
Ri n2 Rh-1 m, nh = n, Re+1 nt and denote ñi = (t- 1)m + n to achieve:
B [[C((
m
1 w 11 p
mn
1 w 1|
2p
p 14e
p The
s'=1
m
wol 112
+
12 ||!ا
w.l1 112 + (1
|| 1|2
n2
(22)
p
p
p
p T4
Next, we set == 2 in Equation 22 to get:
(m + n)o²
[[]]( -1 +
||
wiil + (1
P (p (m + 1)) 1)
p
p
(m n)
and
+
(m + n))
[[[2(wz)] ( (p (m + n)
will + (1
p
p
(m n))
Therefore, the gencralization error is
B[G((z)) = (BEL + (1(((()))
71
(1 +
m
(m - n) - 1) 11w5 - will + (1
2m
m + +1
w|will +
(m + n2)o²
2p
(p
p- (m + n)
Similarly,
[[L](WT)] = (11 -- "1
n
p
Therefore,
B[F(wz)] (wz) E|C, (wT)
17 (1 +
m
w6112
m
(m n))²2
n 2
p (p- (m + n) 1)
p
(m + n) -
- n
30

---

Published in Transactions on Machine Learning Research (03/2025)
A.4 Proof of Theorem 4.2
We start by proving a theorem for the general case:
Theorem A.9. Assume my m2 = m m and n4+1 n. Also denote uz+1 Im and let
0₁ = - p-tm- Considering the continual learner described in Equation 16 when p N + 2, the loss of task at timestep t, can be recursively calculated as:
E[ B1| 1|
(1
1w0 will
| 11 +
mnn
1 w
p
p
p(p n4
1)
==1
+ m 2
1
Ot
| w/r11
2p p
The
(1g 1.
ar n2)22
+ p 12t 1
p
(Tt 72)
+ [,][[l17 -wfll w]11
(23)
Proof. For notation simplicity, denote r : + Using Lemma A.6 and the notation introduced Section A.I
know that:
"ii = (I- P X
Notice that X (px((m+n) is the result of the concatenation of all data points in the memory to the new data for task + 1. Importantly, Xirr contains exactly m columns from task t. Next we can write:
where Pi:r -xx and X (x,
W = (1- W + Σ wo + X
8=1
where Pojs (X, X Xoo Therefore,
B[C((WEFT) = Elllwa7t wi'l?]] [1
= [|
Pol - -o²) + (T (w - w;) + x 21r11111
Pols = P(r))
[||l
(w) - w + (I (w w;)]
- EIIX
(2 Il X E[z1,] 0)
Ell Σ Poll(w - -nf) E[[[(T - (w=)) - m3 + =1r =1r
((I ( Po.
The form of this expression very similar to what we used in the proof of Theorem 3.2 Section A.2.2 The only considerable difference that causes complications the term E[][(T - Pi..)) (w - w )] The reason is that the term wy is no longer independent of Pt.r and it is not straightforward to calculate this expectation. Therefore, avoid repeating the other steps and only focus on finding this term:
31

---

Published in Transactions on Machine Learning Research (03/2025)
| - (u - = -0 [ Pur - w;)l
Notice that w is independent of X,.. Therefore, decompose the second term using Lemma A.3 part (i):
P - w = PI]l (wT - + (1 - -M
P1))( w; )|
P - w - + BII -
-1=))) - wz !1?)
Recall that X, IR' follows the standard normal distribution and independent of (w7 wi) and Pie Due to the rotational invariance property, can write:
Bl] w = PEI (w:( w + (-- p- tm
BBlI(1 - P1:))(77 - w1)
(1 p tm P1 w 11 + n
Billuy wfll? 11
p tm
Therefore, it suffices to focus on E[] P1 - wf)l as the next step. We go back to the definition of w and use Lemma A.6 again. Notice that wy was trained on the samples in the memory from tasks 1 to t - 1 (each task with size m) in addition to the data from task with size Let's build a matrix X1:1 E [=x(((-1)n+t)) to represent all of the training data used in that step. Similarly, build di: and a by concatenating all the label and noise values. Also denote Xojs E (Rx((t1))+t)) with zeros at all columns except the columns corresponding to task s. Then we can write:
w wp= =1 P1--P= Σ Pojorr + x1 x1.
where 81. (I = &1, '8 and Pojs =
Notice that the first Im columns of &1:: exactly X1:: and there are exactly n n-m extra columns X that we threw away before proceeding the task + 1. Let 8, (px(u-m) represent this portion of the training set. In other words, 81.. [Xi: X.]. With this in mind, based on Lemma A.3 part (v), it holds that
X1. = [X -1- P1a) P1a)8.) Xt. (1 (1- (L) - -.
PiuPi.+ = in &T.
[x] 0] 01 Pi.t
and also
PuePoss P1. Pols
Therefore,
= Pil [[][)((w==]]]1
Pia) - wi) Σ Piu Pojo(w; wi) Pre (1ell
Elll
Po. (w - w/) + X 1|
32

---

Published in Transactions on Machine Learning Research (03/2025)
We have previously calculated terms like this in Section A.2.2 Therefore, can write overall:
El 100---10 w;)l = B - - [19] (1
n
E 11 (1: -
n p tm
tm
[ - w |2 Pe wi²) ||a]
m²
= ar - w [| arl
| w3.11
2p(p tm 1)
m +
|ws- - -w,l +
tmo²
tm
==1
Finally, we have
((Ci(WFFT) = Bl|luct w|]l
m
w/ + plp |
77872
|| w. 1|
2p(p
s=1 s'=1
m
wfll 11 + _ wi w E|[[[I P(r))( -wt)ll? +
72
p The
s=1
=
m
(1 (4)
|m0
w 112 + " "1 11?
n
W will
p
p(p The 1)
+ 111
02
|| | w 1|2
2p p
1
p (71 n
8mm
n102
(n - n2)02
+
p 14
p
(Mg n) 1
+ 0x [[[
11
Using the recursive form in Equation 23 one can exactly find the loss of the continual learner in Equation However, the full form is not intuitive, Therefore, we focus on the two-task case instead.
Proof of Theorem 4.2
Proof. We use Equation 23 and substitute o = and T = 2. Therefore, ag = 1 2 and 01 = 1 a For the loss of the first task, we have
E ((CI Fillwr-wit
will = (1 71 =>]l
[| 112
33

---

Published in Transactions on Machine Learning Research (03/2025)
+ (w])] = [|[ W - will"
11 (1 +
m
)1uu - will + +(1
n
F wi
(n + m))
p m
(1
)lwo will + (1
)1
n " I1llf
p
(n + m)
p m
p
And for the second task:
B[C2(WT)] EillwT will
p noo
will + (1
]| 11?
" )| p
(2 = F|||":F w1]l
+
) || -w5l" w1ll 2 (1 - m
w [|
p p - m
p - (m + n))
(1 71 m
+
m
) 1|00 -=2ll + (1
n m
(1
n )| 11
p m
p - (m + n)
p
Finally, we can find the desired metrics:
B[G(wz)]]
(E [R[Li (wz)) + TR/ ([(2(wz)))
2
(2
n m
(p n, - 1) ||
2m
w +
(1
n
)1
12 || p
2p
m
m
and
B[F(wz)] = [w]][ [E[[(WTT]]
1(1 (1 p (n m) )[w will
p (1 " [|w 1
B Bridging Linear Models and Overparameterized DNNs
Recent litcrature has pointed out several connections between linear models and overparameterized DNNs. In this section, we briefly review this connection to highlight the importance of studying linear models and refer the reader to Section 1.2 of Hastie et al. (0022 for broader discussion. We establish the connection using the concept of lazy training regime.
Assune an i.i.d. data as = {(2,, vi) with inputs 2; E Rd and labels y/ E R. Consider possibly non- lonelinear neural network f(::0) Rd R parameterized by e RP. Under certain conditions (Jacot et al. 2018 Allen-Zhu al. 2019 Chizat et al. 2020). including overparameterization the neural network f(::;) can approxinated by its first-order Taylor expansion around the initial parameters f. Furthermore, by supposing that the initialization is such that (==;0g) ss we can write the following linear form:
34

---

Published in Transactions on Machine Learning Research (03/2025)
f(x;0) = f(2;66) - Vof(2:8b) 80) Vof(2z;o)) 8b).
This approximation is still non-linear in but linear in the parameters O, and it implies that in the lazy training regime, the neural network behaves similarly to a linear model where the features are given by the gradients of the network with respect to its parameters. Specifically, the features are the Jacobian matrix 2; == (2)
Additionally, this approximation allows us to define Neural Tangent Kernel (NTK) (Jacot et al. 2018) as (z): 2,) Vaf(z; 06) Ve](zz; 0o); which captures the inner product of the gradients of the neural network with respect to the parameters at different data points. As training progresses, if the parameters remain close to their initialization (lazy training). the predictions of the neural network can be effectively described by linear model in this high-dimensional feature space defined by the NTK. This connection clucidates how overparameterized deep neural networks can exhibit behavior akin to kernel methods, where the NTK serves as the kernel function that determines the similarity between data points.
C Experimental Details
We presented the most important details regarding the dataset and architecture details in Section only present extra information here.
5
We
We used the standard cross-entropy loss for all our models. For the continual learning models, we used a simple experience replay with a uniform sampling strategy in which we optimized the model simultaneously the data for the new task and the previously stored samples in the memory buffer.
We utilized SGD optimizer with learning rate of 0.01, Nestrov, and 0.95 momentum. We trained all our models for 100 epochs. Notice that there exists an epoch-wise double-descent in deep neural networks (Nakkiran et al.) 2021). Therefore, we fixed the number epochs to 100 and mostly explored the model- wise double descent in our work. For simplicity, no extra augmentation or preprocessing was applied to the dataset. All experiments in this paper were repeated at least times, and the average results were reported.
provide the full version of Table 1 with extra details for all four datasets. Notice that in the CIFAR-100 experiments, we used the frozen ResNet-18 backbone and used the width of the MLP layer to control the architecture budget. Similarly, for IN-R and CUB datasets we used a pretrained ViT-B-16 backbone and changed the model's complexity by modyfing the final MLP layers. However, for the PMNIST datasot, an unfrozen ResNet-18 was used and the whole backbone was optimized.
Further Details Table 1
In Tables 2
The effect of varying sample memory budget and architecture memory budget over the average accuracy and average forgetting of a continual learner on the CIFAR-100 dataset. denotes the width of the MLP layers and is the number of samples in the memory for each class. The last task train error for all experiments is zero, meaning that all models operate in the overparameterized regime.
Table 2:
Sample Budget (MB)
Arch.
Avgerage Accuracy (%)
Avgerage Forgetting. (%)
m
k
Budget (MB)
50
122.8
128
0.4
56.620.4
22.55
40
98.3
1605
24.5
70.6
-7.120.1
30
73.7
2329
49.1
71.9
-5.440.3
20
49.1
2885
73.7
66.3
-11.3
10
24.5
3354
98.3
67. 67.440.3
-9.7+0.3
0
0.0
3768
122.9
63.8
-10.8+0.3
35

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: The effect varying sample memory budget and architecture memory budget over the average accuracy and average forgetting of a continual learner on the IN-R dataset. k denotes the width of the MLP layers and m the number samples in the memory for each class. The last task train error for all experiments is zero, mcaning that all models operate in the overparameterized regime.
Sample
Arch.
Avgerage Accuracy (%)
Avgerage Forgetting. (%)
T
Budget (MB)
k
Budget (MB)
10
1204.2
32
0.1
76.1+0.
-3.3
963.4
5249
240.9
76.920.6
-2.00..3
722.5
7520
481.8
77.
-1.40.4
481.7
9263
722.6
78.920.5
-1.040.1
240.8
10733
963.5
78.5
-0.640.1
0.0
12028
1204.4
19.520.6
-20.5
Table 4: The effect of varying sample memory budget and architecture memory budget the average accuracy and average forgetting of continual learner the CUB dataset. k denotes the width of the MLP layers and m is the number of samples in the memory for each class. The last task train error for all experiments is zero, meaning that all models operate in the overparameterized regime.
Sample Budget (MB)
Arch.
Avgerage Accuracy (%)
Avgerage Forgetting. (%)
k
Budget (MB)
5
602.1
16
0.1
87.640.2
-4.5gu.2
4
481.7
3644
120.4
84.110.5
-..72.0
3
361.3
5249
240.9
88.540.3
-1.820.6
2
240.8
6481
361.4
92.0+0.1
-0.1
1
120.4
7520
481.8
91.740.4
-0.4
0.0
8435
602.1
75.8
-12.220.7
Additional Experiments
In this section, we provide extra experiments on DNNs. Section D.1 provides the MTL experiments with single-head and multi-head architectures but with a frozen backbone. Section D.2 offers the results the experiments with unfrozen backbones. Section D.3 we dig deeper into investigating the optimal weights learned by each MLP layer, given that the layers are unshared across the tasks, and we measure the task similarity each layer using the average inner product of the optimal task-specific weights. In Section D.4 investigate the effect of the MLP depth and provide the corresponding train and test curves. Finally, in Section D.5 we offer extra experiments on CL models and demonstrate the effect of label noise on CL models.
D.1 Complementary Multi- Task Learning Experiments
Figures to 13 demonstrate the full training and testing curves for training the model in an MTL setup with multi-head and single-head architectures with different backbones and datasets. As discussed before in the main text, the training error is zero for both cases in large enough models. However, the multi-task learner performs worse in the single-head architecture due to the conflict at the classification layer.
D.2 Mulki-Task Learning Experiments with Unfrozen Backbone
Next, we perform similar experiments with an unfrozen backbone. The results of this experiment are provided Figure 14 The model-wise double descent phenomena are similarly observable when training the backbone from scratch but at different locations for different configurations (refer to Section for details the architecture).
36

---

Published in Transactions on Machine Learning Research (03/2025)
Table 5: The effect of varying sample memory budget and architecture memory budget over the average accuracy and average forgetting of continual learner the PMNIST dataset. denotes the width of the backbone layer, and is the number of samples in the memory for each class. The last task train error for all experiments is zero, meaning that all models operate in the overparameterized regime.
Sample Budget (MB)
Arch.
Avgerage Accuracy (%)
Avgerage Forgetting. (%)
m
Budget (MB)
50
31.3
1
0.1
68.411.6
15.9102
25.0
6.4
83.2
-1..40.4
30
18.8
34
12.8
79. 1..
-16.841.1
12.3
42
19.5
86.3
-9.5
10
6.2
48
25.4
82.4
-14.010.4
0.0
54
32.1
30.8
-71.5+1.2
nol cad.
muRtowak 010
2.2
Thll Eine
Tra: Error
32
225
332
2048
0032
32
532 2048
$302
122 532
2003
8352
32
125
332
2005 8052
Figure 7: Comparing the single-task and multi-task performance on CIFAR-100 with pretrained ResNet- backbone. The top row shows the experiment with the single-head architecture and the bottom row corresponds to the multi-head architecture. The horizontal axis in all plots the width of the ML. layer which is placed on top of the feature extractor. represents the noisy portion of the training that was corrupted by randomly switching its labels.
37

---

Published in Transactions on Machine Learning Research (03/2025)
malboank 2m0
22
23
Thark ETTE
Toot tr
Train
32
130
512
2011
0392
32
120
512
2048
$192
32
120
512
2043
B152
32
130
512
2013
B392
1.0
3.
2043
Figure 8: Comparing the single-task and multi-task performance CIFAR-100 with pretrained ResNet- 50 backbone. The top row shows the experiment with the single-head architecture, and the bottom row corresponds to the multi-head architecture.
muticalk.e 0
mubicakk 02
3.5
13
134
612
2014
8010
912
2018
$197
124
512
2054
6162
120
512
2054
8042
maticunk
02
24
2.5
2,8
2.3
Than Erte
2.4
22
30
2043
2013
Figure 9: Comparing the single-task and multi-task performance CIFAR-100 with pretrained VIT-B- 16 backbone. The top row shows the experiment with the single-head architecture, and the bottom row corresponds to the multi-head architecture.
38

---

Published in Transactions on Machine Learning Research (03/2025)
malisunk 2
2.28
512
2044
00322
502
2048
$332
128
532
2003
B352
52
1.50
512
2045
1052
end
Thal Ener
Tust E310
Figure 10: Comparing the single-task and multi-task performance on the Imagenct-R dataset with pretrained ResNet-50 backbone. The top row shows the experiment with the single-head architecture, and the bottom corresponds to the multi-head architecture.
Hotikol.. 0
muttcssk 00
arglettah..co
Taa Sine
Toat DINE
2014 8040
124
932
2018
$190
128
512
2054
8160
120
612
2054
1040
malicunk
02
2.0
Tiat SINUT
2.3
Than
Figure 11: Comparing the single-task and multi-task performance the Imagenet-R dataset with pretrained B-16 backbone. The top row shows the experiment with the single-head architecture, and the bottom row corresponds to the multi-head architecture.
39

---

Published in Transactions on Machine Learning Research (03/2025)
92
malboukk am0
muttiaale 02
Than Eroe
T:st Bnur
2
32
130
512
2011
8392
32
220
532
2048
$192
32
120
512
2043
B152
32
130
512
2013
8392
Tean Brot
YOu E-14
1.0
30
24
20
512
Figure 12: Comparing the single-task and multi-task performance the CUB dataset with pretrained ResNet-50 backbone. The lop row shows the experiment with the single-head architecture, and the bottom row corresponds to the multi-head architecture.
mubicake -2
mubicskke 0
38
34
arglecust..o
22
DITA
23
0.5
0.5
2
32
320
512
2042
0052
32
122
532
2048
1192
122
532
2043
6152
32
120
532
2043
8052
22
snofetunk.co
2.0
2.2
Thanr Eree
2.5
3.5
2.4
22
5.22
2043
2012
Figure 13: Comparing the single-task and multi-task performance the CUB dataset with pretrained ViT- B-16 backbone. The top row shows the experiment with the single-head architecture, and the bottom row corresponds to the multi-head architecture.
40

---

Published in Transactions on Machine Learning Research (03/2025)
92
80
22
76
8
Than SHN
Thet CIOP
22
11
70
Teet CHEY
10
1.2
10
Figure 14: Comparing the single-task and multi-task performance on CIFAR-100 with unfrozen ResNet-18 backbone with random initialization. The top shows the experiment with the single-head architecture, and the bottom row corresponds to the multi-head architecture. The horizontal axis in all plots is the scale factor that controls the number of filters in convolutional layers. represents the noisy portion of the training set that was corrupted by randomly switching its labels.
41

---

Published in Transactions on Machine Learning Research (03/2025)
D.3 Investigation of the Task Optimal Weights
In this section, analyze the layer-wise optimal weights for each training task. For this purpose, start with a fully shared MTL model. Remember that in our MTL model, frozen backbone is followed by three layers of MLP and a single-head classifier. At cach time, we pick a specific layer and unsharc it across the tasks by reinstantiating a new weight matrix for each task. Then, we optimize the model on the whole training set while ensuring that the unshared layers only see the data of their own task. We repeat this procedure by unsharing only one layer at time have task-specific optimal weights for each layer.
Figure 15 shows the results of this experiment. As demonstrated, the fully shared MTL model has the worst performance, suggesting that weight sharing does not happen effectively in fully shared model. Although unsharing any layer can siguificantly boost performance, the experiments show that an unshared classification head (i.c., multi-head architecture) is the most effective.
To better understand the reason for such behavior, we look the average inner product of flattened task- wise optimal weights. More formally, let W represent the linearized weight matrix at layer I, optimized specifically for task I, where < 1 and S1ST. We compute the following layer-wise average
S T(T 1)
Σ
L.b W
(24)
t=1 t'=t-=
Figure 16 shows the values of this score at different layers for different model sizes. Although the classification layer has less parameters than the other layers (because bonded the number of classcs from one side), the figure shows that it has the most negative score. In fact, the negalive average score shows that the lasks mostly conflicting at this layer and this explains why unsharing the final layer makes more improvements than the other layers.
shared all layers unshared 1ayer unshared 2
2.0
RO
Train Error
unshered layer unshared classiication head
1.5
80
1.2
Text
60
20
16
54
255
1024 4D96
36
256
1024
4096
1.
64
256
1024
4096
16
64
256
1024
4996
Figure 15: Investigating the effect of weight sharing in MTL by unsharing different layers of the MLP layer and training task-specific layers. The horizontal axis is the width of the MLP layer.
0.1
0.0
Average Inner Product
-0.2
-0.3
layer
layer
layer
-0.4
classification head
128
256
512
4096
Figure 16: Comparing the average inner product score (Equation 24 across different layers after unsharing the weights across the tasks. The horizontal axis the width of the MLP layer.
42

---

Published in Transactions on Machine Learning Research (03/2025)
D.4 Investigation of the Depth
In this section, we investigate the elfect of the MLP depth on top of the pretrained backbones. Figure 1i compares different models with varying depths of the MLP. Notice that our theories are based on the assumption of linear model, and therefore, they do not directly predict the behavior as a function of the depth. However, as discussed in Section B the DNN performs similarly to a linear model as long as certain conditions hold. Figure 17 confirms this claim by showing similar behavior for all different models. Another interesting observation in this plot is the impact of depth over the height of the test error peak, which is notable direction for futurr studies.
hidden layers
Nidden layer
80
Niddon layers
Than Eroor
60
redden layers
2.3
1.8
2.0
Text
3.4
20
0.5
1.0
81
12
12B 512
2008
81.52
32
128
512
2048
01.92
32
120
512
2048
01
32
128
512
2048 8192
Figure 17: Visualizing the effect of the depth the performance curves of the multi-head multi-task learner. The horizontal axis is the width of the MLP layers. Notice that simple lincar layer top of the feature extractor can not fully overfit the training samples and has no tunable hyperparameter for increasing the complexity.
43

---

Published in Transactions on Machine Learning Research (03/2025)
D.5 Complementary Continual Learning Experiments
Figure 1 demonstrates full training and testing curves of a CL selup on CIPAR-100 with different replay buffer sizes using pretrained ResNet-18. Notice that the last task training error is zero for both cases in large enough models. Notice that Figure 18 contains both experiments with a single-head and multi-head architecture, which is respectively denoted as taske-inreemental and domain-ineremental learning the CL literature. Additionally, Figure shows a similar experiment with an unfrozen ResNet- 18 backbone.
(sequentaa Ine-kring
2.0
Toank Toun Broce
(mur2-30M)
Forgatii
TostErr
0.5
16
64
255
1024
4230
16354
250
1024
4035
16204
16
04
250
1024
4095
10354
15
54
236
1024
4008
18264
m (neq) arite meeeuninn)
111 103
(1 300
Trenn
1.00
I11 500 (mull- kea))
Forgeiigg oF
Tea Ererr
13
ThstLo05
0.775
0.50
250 1024 4050 15114
15
64
256
1024
4066 10384
15
04
255
1224
4096
10200
15
14
255
1024
4092
TR264
Figure 18: Visualizing the CL performance CIFAR-100 with different values of memory budget denoted The horizontal axis is the width of the MLP layer. The top row corresponds to the experiments with single-head classifier and the bottom row shows the multi-head architecture.
1.4
(oecuantaal
3.5
80
100
100
2.5
bank Tran
200 (mubitae))
2.0
Teal Exter
0.6
1.6
0.5
0.0
0.0
Figure 19: Visualizing the CL performance on CIFAR-100 using an unfrozen ResNet-18 backbone and a multi- head classification head. The plot shows the performance for different values of memory budget denoted m. The horizontal axis is scale factor that controls the number of filters in the ResNet-18 backbone.
Finally, we present the experiments with different noise levels in Figure 20 As our theories suggest, the error peak intensified with stronger noise.
E Limitations and Future Work
In this paper, we presented theoretical framework for understanding multi-task overparameterized lincar models which is novel and unexplored before. The strength of our work is that we offer exact theoretical closed-form results rather than asymptotic results or generalization bounds for an MTL setup. However, there are also limitations in our work, which we will address in this section.
Our theoretical analysis is based on several assumptions and simplifications to derive closed- form expressions. For instance, assume i.i.d. Gaussian features, the existence of optimal task vectors and additive noise
44

---

Published in Transactions on Machine Learning Research (03/2025)
2.00
1.75
2.0
1.50
Tean
10
0.50
0.3
1.5
0.25
0.00
1.0
Figure 20: Visualizing the CL performance of multi-head models using memory size m = 50 on CIFAR-100 with different levels of noise denoted as a. The horizontal axis is the width of the MLP layer.
in our linear regression setup, which simplifies the mathematical analysis but may not fully represent the complexity and variability of real-world data. These assumptions might limit the generalizability of our results to more complex non-Gaussian data distributions commonly encountered in practical applications.
The next potential limitation is that our analysis is focused on linear models. Although we connected the linear models and DNNs in Appendix B but the connection holds under certain assumptions which may be violated in different applications. Besides, we emphasize that the goal of this work not to make formal claims regarding the performance of multi-task DNNs, rather, we tried to qualitatively connect our understanding of linear models the observations derived experimentally from DNNs. We believe that knowing the trade-offs between different system specifications, such as the model size and number of training samples, can provide important guideline when deploying practical models.
Despite all these limitations, we believe that our work should be evaluated in line with the recent ad- vancements in theoretical investigations of overparameterized models. Additionally, we emphasize that the theoretical tools developed in the proofs section are valuable and novel compared to the related prior theoretical works in the field.
At this time, our theoretical tools for fully understanding overparameterized DNNs are limited and therefore, one possible direction for future work is to focus more specifically on these models. In the context of multi- task learning, it is important to understand what does task similarity mean in deep models operating over complex datasets. Remember that we defined the task similarity using the distance between the optimal task vectors, an assumption which is not necessarily well-defined for DNNs with high-dimensional weights. Therefore, it is essential to provide a better definition of the task similarity and then connect to the double descent: behavior of DNNs.
F Infrastructures and Data Availability
We mainly used PyTorch Paszke et al. 2019 for our implementations and datasets are freely accessible (Imagenet-R (Hendrycks al. [2021). CUB Wah et al.] 2011), CIFAR-100 (Krizhevsky al. (009), and MNIST (Deng 2012)). All experiments in the main text. reproducible single NVIDIA 2080 TI GeForce GPU.
45


Reviews:
[{"review_id": "4GLG3SM1Ev", "reviewer": "Reviewer_Nz8y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is mostly theoretical but nice experiments were carried out to support the theoretical findings.\n- The paper offers a rigorous analysis of MTL in the studied setting, revealing several factors that effect the generalization and transfer.   \n(I cannot assess how novel this part is.)\n- The paper offers one of the first theoretical results on the effect of the replay buffer in CL.   \n(a similar CL analysis has already been done in [Lin et al. (ICML 2023)]; the novel part here is the replay part.)\n\n## Weaknesses\n- The main weakness is that under the investigated data and model assumptions, the analysis becomes uninformative when $p\\to\\infty$, which is generally the more interesting regime. For instance, Theorem 4.1 simply gives a generalization bound of $1$, while a naive solution of $w=0_d$ would yield the same generalization. This is also apparent from Figure 1.  \nMoreover, many times the authors discuss phenomena that only occur above that generalization \"floor\". That is, when the generalization error is already above the naive solution's error.\n(The authors seem to have taken a similar analytic approach as a previous CL paper which also suffer from this weakness.).  \nHowever, I still find the paper here interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- Characterize the generalization error of a linear model in a specific multi-task learning (MTL) regression setup.\n- Use the analysis to understand the effects of different factors like overparameterization (in a double descent spirit) and task similarity.\n- Extend their analysis to a continual learning (CL) setup, analyzing similar factors, but also factors like the size of the replay buffer, which is not fully understood yet in existing CL literature.\n- Show empirical results with deeper models, generally agreeing with the analysis on the linear models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major changes\n1. The authors use the term \"regularization methods\" in a way that I find misleading.  \nGenerally, regularization methods solve $\\min \\left\\Vert X_{t}w-y_{t}\\right\\Vert^2 + \\lambda \\left\\Vert w-w_{t-1}\\right\\Vert^2_{W}$ where $W$ is some PSD weighting matrix.  \nHere, the authors use isotropic regularization (which is fine), but also implicitly assume that $\\lambda \\to 0$. This yields Eq. (13), and is implicitly induced by plain, unregularized SGD [Evron et al. (2022)]. It is of course an interesting setting to investigate (appearing in many recent work), but is *not* what people usually call a \"regularization method\". The terminology needs fixing.\n\n## Questions\nI would like the authors to help me understand the following issues.\n1. In the replay Eq. (12), why is the second constraint (on the replay buffer) valid in a noisy setting? Doesn't it heavily depend on the overparameterization level?\n1. The authors say on Page 10 \"unsharing the classification head is the most effective approach\". I'm not sure I understand how this is possible (even technically).\n1. On Lemma C.4 and throughout the appendix, the authors use $(X_1^\\top X_1)^{-1}$. However, generally this matrix isn't necessarily invertible. Do the authors mean to state their claims w.h.p. or a.s.?\n1. How does Term G1 in Eq. (16), when $m=0$, settle with Term G2 in Eq. (10) of Lin et al. (when $T=2,\\sigma=0$)? It seems they yield different coefficients.\n\n\n## Minor changes\n1. At the end of page 3, the authors cite Evron et al. (2022) as assuming a noisy data model, while that paper actually assumes a noiseless data model.\n1. In the second paragraph on Page 4: should probably use \"choose\" instead of \"chose\".\n1. Some of the vectors are bold (e.g., $\\mathbf{z}_t$, while others are not (e.g., $w_t$).\n1. In Figure 2a: It is better to use `semilogx` with the actual x-values rather than manually taking the $log$.\n1. The figures would benefit from briefly stating their message (conclusion) at their end.\n1. When discussing overparamterization, the authors refer to Goldfarb & Hand (2023). Two recent works [Goldfarb et al. (ICLR 2024), Guha and Lakshman (ICML 2024)] should also be referred to and discussed briefly in that context.\n1. Something is wrong in the parenthesis of Eq. (15).\n1. Lemma C.4 confused $p_1$ with $P_1$.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XBN2TjH648", "reviewer": "Reviewer_s2oY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a well-written paper with clearly stated major components. The theoretical insights are interesting and valuable. While most conclusions are derived from a linear model, they provide meaningful intuition and serve as an initial step toward understanding more complex, realistic models. The authors also acknowledge existing work extending linear models, such as neural tangent kernel models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides theoretical insights into multi-task learning (MTL) and replay-based continual learning (CL) in overparameterized settings, using linear models as proxies for deep neural networks. It derives explicit formulas for generalization error and knowledge transfer, highlighting the impact of model complexity, task similarity, and dataset size. The study also examines the role of memory buffers in CL and validates the findings through empirical experiments, demonstrating their relevance to deep learning models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. $P_{0|t}$ defined on Page 18 is not a projection. Using the notation $P$ may be misleading; consider changing it or adding a clarifying remark.\n\n2. In Lemma C.6, $P$ should be $X (X^T X)^{-1}X^T$ instead of $X (X^T X)^{-1}X$.\n\n3. In the proof of Lemma C.7 (iii), $P_1X_2$ and $(I - P_1)X_2$ are not independent (e.g., when $P_1X_2 = 0$, then $(I - P_1)X_2 = 0$). The correct statement should be that $P_1X_2$ is uncorrelated with $(I - P_1)X_2$, which is sufficient for applying $E[ab] = E[a]E[b]$. If this is the case, other places, such as the last line of Page 23 (\"$P_1X_2$ and $(I - P_1)X_2$ are independent.\"), should also be revised accordingly.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SrG91Gbgys", "reviewer": "Reviewer_5YYX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper offers theoretical analyses into overparameterized linear models in MTL and CL for understanding generalization and knowledge transfer.\n- It validates its theories through experiments on real-world datasets and architectures, showing applicability to deep neural networks.\n- The study also shows trade-offs in memory size and model capacity in continual learning.  \n\n### Weaknesses \n- The novelty compared to existing generalization bound results is not explicitly stated. The paper claims that they derive the first generalization-bound results for MTL with linear and overparameterized models. How about simply applying existing bounds to MTL settings (for example Hastie et al. (2022) cited in the paper), and what advantage do the proposed results have over the naive ones? How do the results differ from Wu et al. (2020)? \n- The implications of the theoretical results are not clear. How well does the bound track generalization errors in practice? and Do they suggest some algorithmic design based on the theoretical results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper develops a theoretical framework to analyze the generalization error and knowledge transfer in overparameterized linear models used for MTL and replay-based CL. It explores how model size, dataset size, and task similarity impact these metrics. Notably, it characterizes the interplay of these parameters in both collaborative and conflicting task scenarios.\n\n- The study extends the theoretical analysis to replay-based continual learning models, focusing on how buffer size and model capacity influence forgetting rates and generalization error. This helps understand the effectiveness of memory buffer strategies.\n\n- This paper conducts extensive experiments using both linear models and deep neural networks, on CIFAR, ImageNet-R, and CUB-200 datasets. The paper demonstrates that its theoretical insights generalize to more complex systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The work shows a peak exists in the generalization error curve of the multi-task learner when increasing model complexity. I found this interesting. It would be great if the peak could be explicitly explained after presenting theoretical results. \n- There are many findings and implications. It would be great to clearly outline them after the introduction, and make them parallel in the theoretical results and experimental results. \n- In Section 4, the paper mentioned that \"the continual learner can sometimes outperform the MTL baseline even when tasks are collaborative, especially at the multi-task learner’s interpolation threshold.\" Is there any connection to the theoretical results? \n- There are many experimental results. It would be great to provide a summary at the beginning of the experiment section. \n- How does this paper define task similarity/dissimilarity? It would be great to provide a mathematical definition.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RMqzrytVUl", "reviewer": "Reviewer_f14z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper provides a detailed and unified characterization of Multi-Task Learning (MTL) and Continual Learning (CL) under various settings using linear models. The paper is well written and organised, making it enjoyable to read.\n\nWeaknesses:\n\n1. It would be beneficial to conclude practical insights. For instance, stating that \"larger models with greater capacity are less vulnerable to forgetting\" is not particularly surprising.\n2. In Equations (12) and (13), why must the equality constraints be strictly satisfied? I believe there may be some noise in the data.\n3. Better to discuss some recent works, e.g. [1].\n\n[1] Online Continual Learning via Logit Adjusted Softmax, TMLR'24", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates Continual Learning (CL) and Multi-Task Learning (MTL) within the context of overparameterized linear models, developing theoretical results that describe how various system parameters influence generalization error and knowledge transfer. Empirical evaluations demonstrate that these theoretical findings apply to deep neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I have not thoroughly examined the theoretical results, as I am not an expert in this area.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work makes a solid contribution that, among other things, shows theoretical results exhibiting the effect of replay buffer size in continual learning. The experiments are interesting. The authors show that a double descent phenomenon occurs in the continual learning setting. This work is a welcome contribution to TMLR.
Decision: Accept (poster)