Is multitask learning all you need in continual learning?

Authors: Giulia Lanzillotta, Mandana Samiei, Claire Vernade, Razvan Pascanu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: lifelong learning, multitask learning, continual learning

Abstract:
Continual Learning solutions often treat multitask learning as an upper-bound of what the learning process can achieve.  
This is a natural assumption, given that this objective directly addresses the catastrophic forgetting problem, which has been a central focus in early works. However, depending on the nature of the distributional shift in the data, the multi-task solution is not always optimal for the broader continual learning problem. In this work, we draw on principles from online learning to formalize the limitations of multitask objectives, especially when viewed through the lens of cumulative loss, which also serves as an indicator of forward transfer.
We provide empirical evidence on when multi-task solutions are suboptimal, and argue that continual learning solutions should not and do not have to adhere to this assumption. Moreover, we argue for the utility of  estimating the distributional drift as the data is being received and show preliminary results of how this could be exploited by a simple replay based method to move beyond the multitask solution.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729631074004

thread_type:
official_review

<thread_object>
creator: reviewer_dyh5

<content>
summary:
The paper defines "average life long error" as the average error the continual model had on the task it was trained on. Then, the paper compares between single-task and multi-task agents from that perspective, showing that either agent can achieve lower error depending on the specific task sequence. The paper proves that in a convex under-parameterized case, when training the agents on each task for a long enough period of time, the single-task agent actually reach better results. The paper continues with an empirical study, showing similar results in a range of simple toy experiments.

strengths:
Originality: The paper introduces a novel metric, which allows for original analyses. The analyses give birth to elements like "instability", which can help in giving intuition to the difficulty of task sequences.

Clarity: The paper is written in a clear way, and it is easy to follow the presented ideas. The ideas presented are simple and intuitive, and the motivation behind them is generally clear.

Significance: Studying the reliance of continual learning algorithms on multitasks solutions is interesting and with a value to the community. The paper does make you question this point, which is not often addressed in the literature.

weaknesses:
The "average lifelong error" sums the empirical risks of models trained sequentially, but only on the test data that comes from the distribution of the task that the models are trained on at that time. This effectively ignores the "continual" aspect of the models: in continual learning, we want to train a model first on one task, then on a second task, and expect the model to perform well on both tasks simultaneously.

Using the "average lifelong error" metric places multi-task agents at a disadvantage. While single-task models are evaluated solely on the data they are trained on, multi-task agents are also trained on unrelated data, which is not part of their evaluation. Consequently, unless the additional data seen by the multi-task model is highly correlated with the task at hand (as in the example shown in Figure 2b), it is expected that the single-task model will train faster, converge more efficiently, and achieve a better "average lifelong error" compared to the multi-task model.

Although this outcome seems intuitive given the experimental setup, the paper overemphasizes this result, as it claims that so far the multi-task agent served only as an upper bound, and this shows that it is not always true. Therefore, the manuscript asserts that "continual learning solutions should not and do not have to adhere to multitask objectives". I find this conclusion overstated: The metric is built in such a way that is skewed against multi-task learning, which is bound to fail it given the suggested metric, which does not capture the essence of continual learning. In all the experiments in the paper, when testing the total accuracy across all tasks, the multi-task agent does overperform the single-task agent.

That said, the proposed "average lifelong error" does provide valuable insight. Specifically, if a multi-task agent performs better than a single-task agent, it indicates that the unrelated data encountered by the multi-task agent is beneficial for training, and the method can serve as a useful measure of data similarity. The authors acknowledge this in the paper, but I believe this point should be emphasized more, while the broader claims about continual learning should be moderated. As a result, the quality of the paper drops (as the claims are not fully supported), and the significance of the findings is much more limited (as this point is much weaker than the general claims the paper suggests).

Nevertheless, I appreciate the soundness and depth of the analyses presented. Including both theoretical and empirical sections strengthens the work, even if the experiments are conducted on small-scale toy problems. However, given that the analysis focuses predominantly on the "average lifelong error," the insights regarding continual learning are somewhat limited.

questions:
What is the motivation behind ignoring the risk over the previous task in the "average lifelong error"? In which situations would we want models that perform well throughout the entire learning, ignoring the final performance? If the performance on past tasks is forgotten, what intuition can be gained for continual learning, using this measurement?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730119604662

thread_type:
official_review

<thread_object>
creator: reviewer_cvzj

<content>
summary:
This paper challenges the assumption that multi-task training is the upper bound for continual learning solutions that learn on a stream of such tasks. They formalize the limitations of multitask objectives and show cases where the multitask optimal solution is different from the continual optimal solution, both with a toy example and with a modified version of a "real-world" dataset.

strengths:
- The authors propose to challenge an assumption that is often made in continual learning, bringing that topic into discussion. Challenging this assumption is reasonable and needs to be backed up preferably by both simple examples demonstrating why the assumption does not hold in some cases, and empirical results showing that this assumption can also be challenged in real world problems. The paper follows that structure which is good.
- The paper is well written and quite easy to follow.
- Many benchmarks are chosen for empirical evaluation

weaknesses:
- **W1**: The toy setting is too simple and does not accurately reflect what could occur in the wild. This setting assumes something similar to a label switch while keeping the same head, which means that for instance an object would have to be recognized as something and then as something else, leading to an impossible solution for the multitask loss (even in the offline learning case this loss could not learn anything but overfit since it would be asked to learn from contradictory signals). I think a better example should show that even if there is a satisfying solution for multitask loss, learning using this loss does not result in an upper bound for the CL method (at least in term of learning efficiency)
- **W2**: As it is right now it is not clear why you split the empirical evaluation in two parts, one part using the CLEAR dataset and MD5 and another part using the CIFAR10 permuted. This needs to be explained more in details why these two parts are needed and what do you want to show in each part.
- **W3**: The main weakness of the paper in my opinion is that it is too disconnected from the rest of the online continual learning works. It is true that is online learning most metrics that you present are used and people care more about the rapidity of adaptation rather than the retaining of previous knowledge. But in most online continual learning  works, that is not the case, and metrics such as average accuracy, average anytime accuracy (area under the curve of AA) are used way more. So you need to justify why you focus only on these metrics. So far the only continual learning paper I know that used these is a paper on the CLOC dataset, where it is kind of justified to look at the adaptation metric since the knowledge needs to be "updated" and there is no need to "retain" previous knowledge. But in many of the benchmarks that you present, the retaining of previous knowledge is key to performing on the test set. So it is unfair to present the multitask baseline as "under performing" compared to the CL solution under these metrics that the MT baseline is not suppose to be the upper bound of. **In Short** , I agree that these metrics are important, but more classical metrics should also be reported and the advantage of CL methods (could be CL methods that use infinite memory) on these metrics should also be shown.
- **W4**: You claim in the paper that most CL methods use the multitask baseline as an upper bound, which I agree on. However, you also claim that most CL methods use the ERM objective. I think this is not entirely true. First of all, when there is no replay it is not the case most of the time. Secondly, even when replay is used, in general it is not used to get a precise estimation of the ERM objective because of the different weightings applied to the memory batch and current batch. Most CL methods would just draw one batch from the current task and one batch from  the memory and sum their loss to get the training loss. This results in giving more weight to the current task data compared to the memory when num_task > 2,  and it results in a different objective than the ERM. I think this part of the story could be tuned down a bit.

questions:
- In the CIFAR10 benchmark used for empirical evaluation, you say that you vary the permutation size of the two datasets, but it is not explained what entries are permuted, do you chose a random pixel set to apply the permutation on ?
- I think there is a mistake in Figure 1, caption should say "On the right, ... while on the left, reverse is true"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730593184908

thread_type:
official_review

<thread_object>
creator: reviewer_jwtx

<content>
summary:
This paper studies CL under a new metric borrowed from Online Learning, called dynamic regret. It is shown that under this new metric, matching a multitask learner, which is a common goal in the literature, might be suboptimal with respect to dynamic regret. Specifically, performance of a single task learner is compared  to that of a multi task learner in a linear model to gain insight into when multitask learner is optimal. Empirical analysis supporting the theoretical findings is provided.

strengths:
This paper is studying a question that is important to the community, and is well motivated. The arguments are laid out mostly clearly and are easy to follow. The experiments are extensive.

weaknesses:
I think the main caveat of this work is the underlying assumption that the risk of the defined multi task agent, which sequentially trains on tasks starting from the previous solution, converges (with number of steps $h \rightarrow \infty$) to the risk of true multi-task solution which is a minimizer of average risk of tasks seen so far.  Using the notation in equation (2),  the claim is that $\Delta_T^{MT} \rightarrow 0$. This assumption is not explicit, it is mentioned  in line 196, that it holds in convex settings. Looking in the appendix section B.1.2, however, it seems that MT agent defined in the linear convex setting takes gradient steps with respect to an objective that takes into account all tasks simultaneously. So this does not match the description of the MT agent given in line 169, which I think needs to be clarified. 
My understanding is that it is not easy to match the performance of a true multitask learner (that minimizes error on all tasks simultaneously ) while learning continually. 
The empirical analysis section is a little bit hard to follow. It is not always easy to follow which part of the narrative each figure/paragraph supports. Some examples:


 - Table 3 : not sure what to expect by looking at the number of tasks. What is the hypothesis here?
 - Table 1: it seems that $v_{agent}$ is tracking error while $O_{agent}$ is tracking accuracy.

Description of the algorithm Selective Replay is missing from the main text.

questions:
It would be great if the authors could explain the discrepancy between the linear MT agent and then one that trains continually on one task at a time. 

Suggestions:

- I think moving equation 19 to main text and moving equation 4 would be helpful. Is there a $\Sigma_x$ subscript missing from equation 4?
- Include definition of SR in the main text (I could not find it in the appendix)
- line 457: says instability is higher for  PC-16.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730802074767

thread_type:
official_review

<thread_object>
creator: reviewer_8lkj

<content>
summary:
This paper challenged a long-standing assumption in continual learning (CL): multi-task learning (MTL) is the upper bound for CL. Authors found that MTL is not always the upper bound for CL especially in highly non-stationary environments or long sequences. To explain their findings, theoretical results showed that the single-task system is more suitable in a volatile environment. Experiments were conducted to confirm the hypotheses and theoretical results in both synthesis and real-world environments.

strengths:
This paper has several strengths:

- This paper questions a popular but underexplored assumption in CL: is MTL always an upper bound for CL system? The authors show that this is not true in highly complex environments both theoretically and empirically. Answering this question allows us to understand when we should ignore the MTL results in a benchmark and explain why several CL methods yield better results than MTL.

- The authors conducted a comprehensive experiment to verify their hypothesis. The empirical results verify their theoretical results.

weaknesses:
Despite these strengths, my main concern is about the contribution of this work:

- Several parts of the paper need more clarification for smoother reading and understanding. I struggled during reading Sections 3 and 4 with several notations that were not fully explained. E.g., the $\theta^*$ in Eq.4. Many typos in paragraphs of the main paper such as in line 370. I recommend authors carefully revise the main paper during the rebuttal process.

- Since the DL models are mostly overparameterized, the theoretical results only consider the linear models in a strictly convex setting, limiting the contributions of this work. I wonder what happens if we add the regularization term in the loss function as indicated in the discussion part.

- Although the authors pointed out that there are some cases that the MTL is not as good as ST, I wonder: is there any recommendation, signal, or measure for practitioners to recognize these cases before training and estimate the instability? it would make this work more solid. 

- Despite in Sec.3 and Sec.4, the authors emphasize that the setting of this paper is online learning. However, in the experiments, authors use h = 3000, 6000,... In my opinion, it resembles the offline continual learning setting when each task is trained for several epochs not a single one like in online continual learning. Is there any explanation for this?

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732043199307

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment to all reviewers

comment:
Overall, despite the low scores, the general sentiment of the reviews is constructive. All the reviewers' have raised very valid points. As highlighted by the reviews, our work was lacking in presentation, and we believe that the paper quality has increased dramatically by incorporating the reviewers' feedback. Thus, regardless of the scores, we are very grateful to the reviewers and their work. In the past days, we have put a lot of effort into updating the manuscript and we would kindly ask the reviewers to have a (quick) look at the new version uploaded on openreview.  Where possible, we have highlighted the changes made in blue. In the personalized responses, we will refer to the changes which concern the specific question of the reviewer. However, we recommend a complete read of the new version, as we believe that the overall exposition, flow, coherence, and quality have been significantly improved. 

What follows is a list of the **main changes applied**: 
1. We have modified the text, especially in Sections 1 to 4, increasing clarity, attending especially to the points of confusion raised by the reviewers. 
2.  In particular, we expanded on the motivation behind this work, addressing the questions and doubts of Reviewer Dyh5.
3.   Appreciating the suggestion of Reviewer cvzj, we included a discussion of existing online continual learning metrics and the link between this work and that line of literature, which is definitely relevant to our paper. 
4.   Given the doubts raised on the connection between MT and CL objectives - which is essential to this work - we expanded on this background topic in Section 2 in the main paper and with a dedicated section in the Appendix. 
5.   We greatly improved the theoretical analysis, overall **adding new results and stronger claims** in Section 4 and the Appendix. In particular, we have worked out a matching lower bound for $\Delta_T$. This allows us to provide tight bounds on $\Delta_T$ in general, and to define more precisely the nature of the dependency on $h$. Moreover, we have expanded our derivations to cover the overparametrized case. In addition to expanding it, we have also cleaned the (previously sloppy) exposition of the theoretical results.
6.   We have improved our toy settings, prompted by Reviewer cvzj (and updated accordingly the math in the appendix) and included simulations to estimate the critical task duration. We believe now the toy settings make a much clearer exemplification of all the factors which we have isolated in the theory.
7.   We have greatly improved the exposition of the experimental section. Following the suggestion of Reviewer cvzj and JWtx, we have added a description of the structure of the experimental section at the beginning, and we have placed the tables close to the respective paragraph. We have also changed all the supervised learning metrics from errors to accuracies to improve the interpretability and align better with the rest of the literature. 

We hope that our efforts will be noticed, and that - if the reviewers also find the paper improved - they would promote acceptance to the conference. Especially considering that all the feedback pointed out problems on the presentation we believe this work is not lacking any substantial component and is already complete.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732785442626

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final revised manuscript

comment:
Dear all, given that the deadline for the revision period is today, we have uploaded a new revision with minor changes:   
- we have moved the discussion of limitations to a separate section in the Appendix, in response to the concerns of Reviewer Dyh5 
- we have included in the limitations some considerations regarding the MT objective,  as suggested by Reviewer JWtx

We thank all the reviewers for the input.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the problem of episodic continual learning, in which a neural network aims to learn new predictive tasks throughout learning without forgetting original tasks. The primary contribution of this paper is theoretically and empirically demonstrating that the multitask learning setup - which is often assumed to be "an upper-bound of what the learning process can achieve" - can be surpassed by a single-task learner under the online learning metric of dynamic regret.

In many ways, the paper is well-executed: it is written clearly, contains sound theory and experiments backing up the theory, and challenges a long-held assumption of continual learning. Nevertheless, this paper could benefit from an additional round of reviewing to address remaining concerns about significance and scope. The new metric introduced for continual learning can be perceived as being biased against multitask learning, so the paper would benefit from a comparison to other metrics more commonly used by the community. Moreover, the primary finding - as it is presented in the paper - can be interpreted as a simple refinement of the well-established observation that multitask learning is suboptimal when tasks are not sufficiently related.

Shifting the focus of the claims will significantly strengthen this paper, especially since the claims revolve around "a topic of ongoing debate within the CL community."