Optimal Task Order for Continual Learning of Multiple Tasks
Authors: Ziyan Li
Naoki Hiratani
Venue: ICML 2025 poster (2025)
Keywords: Lifelong Learning
Curriculum Learning
Primary area: theory->deep_learning
TL;DR: Continual learning is most effective when adjacent tasks are dissimilar and the final tasks are representative of all learning tasks.

Abstract: Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose a strategy for optimizing it. Leveraging a linear teacher-student model with latent factors, we derive an analytical expression relating task similarity and ordering to learning performance. Our analysis reveals two principles that hold under a wide parameter range: (1) tasks should be arranged from the least representative to the most typical, and (2) adjacent tasks should be dissimilar. We validate these rules on both synthetic data and real-world image classification datasets (Fashion-MNIST, CIFAR-10, CIFAR-100), demonstrating consistent performance improvements in both multilayer perceptrons and convolutional neural networks. Our work thus presents a generalizable framework for task-order optimization in task-incremental continual learning.

Optimal Task Order for Continual Learning of Multiple Tasks
Ziyan Li"
Naoki Hiratani?
Abstract
et al.; Kontogianni et al., 2024).
Continual learning of multiple tasks remains a major challenge for neural networks. Here, we investigate how task order influences continual learning and propose strategy for optimizing it. Leveraging linear teacher-student model with latent factors, we derive an analytical expression relating task similarity and ordering to learning performance. Our analysis reveals two principles that hold under a wide parameter range: (1) tasks should be arranged from the least representative to the most typical, and (2) adjacent tasks should be dissimilar. We validate these rules on both synthetic data and real-world image classification datasets Frsshoo-MNIIST CIFAR-10. 10, CIFAR- 100), demonstrating consistent performance im- provements in both multilayer perceptrons and convolutional neural networks. Our work thus presents generalizable framework for task-order optimization in task-incremental continual learn- ing.
One important yet relatively underexplored aspect of con- tinual learning is task-order dependence. Previous work has revealed that the order in which tasks are presented can sig- nificantly influence continual learning performance and also explored various approaches to optimize task order (Lad et al., 2009; Pentina et al., 2015; Guo al., 2018; Bell & Lawrence, 2022; Lin al., 2023; Singh et al., 2023). How- ever, we still lack clear understanding on how ordering of tasks influences the learning performance and how to order tasks to achieve optimal performance. Figure 1 illustrates this problem using a continual binary image classification example, where a neural network is trained on three tasks: (Cat vs. Ship), (Frog vs. Truck), and C (Horse vs. Deer). Learning one task can influence performance on the others in a complex manner (Figs. la and 1b). Consequently. in this example, the C-B-A task order achieves higher average performance after training than the A A-B-CC C or- der (Fig. Ic). The goal of this work is to understand this task-order dependence in continual learning.
Task-order optimization requires some amount of knowl- edge on all tasks beforehand, making it infeasible in strictly online learning setting. Nevertheless, it remains highly relevant for many learning problems. One scenario is when data acquisition and training need to be conducted in parallel, which may occur in the training of self-driving algorithms (Verwimp et al., 2023) or medical image analysis (Kumari et al., 2023). In this setting, it is beneficial to first collect a small pilot dataset across all underlying tasks and then determine the optimal order of data acquisition and training to maximize knowledge transfer while minimizing forgetting across tasks. To demonstrate the potential appli- cability of our theory to this problem, we provide numerical evidence that, in continual visual recognition benchmarks, an optimal task order estimated from just 1% of the data significantly outperforms a random task order.
1. Introduction
The ability to learn multiple tasks continuously is a hallmark of general intelligence. However, deep neural networks and its applications, including large language models, struggle with continual learning and often suffer from catastrophic forgetting of previously acquired knowledge (McCloskey & Cohen, 1989; French, 1999; Hadsell et al., 2020; Luo et al., 2023). Although extensive work has been done to identify when forgetting is most prevalent (Ramasesh et al.; Lee et al., 2021) and how to mitigate (French, 1991; Robins, 1995; Kirkpatrick et al., 2017: Shin al., 2017; Serra al., 2018; Rolnick et al., 2019), it remains unclear how to pre- vent forgetting while simultaneously promoting knowledge transfer across tasks (Ke et al., 2020; Lin et al., 2022; Ke
Moreover, in robotics applications (Lesort et al., 2020; Ibarz et al., 2021), switching between tasks often involves phys- ically rearranging objects around the robot, which is both consuminn and labor-intensive As a result, switching tasks on a trial-by-trial basis often infeasible, necessitating block-wise training. In this scenario, optimizing task order could help maximize average performance across all tasks.
'Department of Physics, Washington University in St Louis, St Louis, USA 'Department of Neuroscience, Washington University St Louis, Louis, USA. Correspondence to: Naoki Hiratani <hiratani edst..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Optimal Task Order for Continual Learning of Multiple Tasks
b
Classiiication accuracy
Classification accuracy
1.0
0.8
Final accuracy
2.8
C-BBAA
A-B--C
0.6
0.6-
8.6-
2.4
A: Cat vs Ship 0.2
Hosse vs Deer B: Frig vs Truck A: Cat Ship
Frog vs Truck
02
0.2
Horse Deer
Epoch
15
Epoch
Order a
Order b
Figure 1. Schematic figure of the task-order dependence. a, b) Continual learning of binary classification with two different task orders. <) Average test accuracy on the three classification tasks at the end of learning under task orders depicted in panels a and b. Error bars represent the standard error of mean over random scods.
Similar constraints arise in designing of machine-learning based teaching curricula for schools or professional training where learners need to study multiple subjects sequentially (Rafferty et al., 2016; Singh al., 2023). Furthermore, even in more traditional continual learning task, if the current task creates unfavorable conditions, systems can postpone its learning to a more suitable time. Understanding how the order of tasks impacts learning can also serve as tool for predicting the difficulty of online learning given a data stream. Lastlyy, large language models are typically trained in an online fashion because the size of the training corpus is so vast that multiple epochs of training over the entire dataset is infeasible (Hoffmann et al., 2022: Chowdhery et al., 2023). In such cases, how the corpus is organized for training can significantly impact learning efficiency, sup- porting the importance of optimizing task/corpus order.
the path length in the task dissimilarity graph outperforms one that minimizes this path length (max-path rule), consis- tent with previous empirical observations (Bell & Lawrence, 2022).
We illustrate these two rules by applying them to tasks with simple similarity structures forming chain, ring, and tree graphs, revealing the presence of non-trivial task orders that robustly achieve the optimal learning performance, given graph structure. Moreover, we apply these rules to con- tinual image classification tasks using the Fashion-MNISTT, CIFAR-IO, and CIFAR-I00 datasets. We estimate task simi- larity by measuring the zero-shot transfer performance be- tween tasks, and then implement the task-ordering rules based on these estimates, Our results show that both the periphery-to-core rule and the max-path rule hold robustly in both multilayer perceptrons and convolutional neural networks. Moreover, using ~ 1% of the data for the task similarity and order estimation was sufficient to achieve a significant improvement over random ordering. This work thus provides a simple and generalizable theory to task-order optimization in task-ineremental continual learning.
To explore the basic principle of task order optimization, here we analyze the task-order dependence of continual learning using linear teacher-student model with latent factors. First, we derive analytical expression for the average error after continual learning as a function of task similarity for an arbitrary number of tasks. Our theory shows that this error inevitably depends on the task order because it is a function of the upper-triane upper-triangular components of the task similarity matrix, rather than of the entire matrix.
2. Related Work
The effects of curriculum learning have been extensively studied in the reinforcement learning (RI.) literature (El- man, 1993; Krueger & Dayan, 2009; Narvekar et al., 2020), However, these studies primarily focus on learning single challenging task by sequentially training on simpler tasks, leaving open the question of how to design a curriculum for learning multiple tasks of similar difficulty. A limited number of works have explored task-orde optimization for continual/ifelong learning across multiple tasks by contrast.
We then investigate how the similarity between tasks, when placed in various positions within the task order, affects the overall error. Through linear perturbation analysis, we find that the task-order effect decomposes into two factors. The first absolute order dependence: similarity between two tasks influences the error differently depending on whether these tasks appear near the beginning or near the end of the sequence. We demonstrate that when tasks are average positively correlated, the least representative tasks should be learned first, while the most typical task should be learned last (periphery-to-core rule). The second factor is relative order dependence: the effect of task similarity on the error differs depending on whether two tasks are adjacent in the sequence or far apart. We show that task order maximizing
Lad et al. (2009) demonstrated that ordering tasks based on pairwise order preferences can lead to better classifica- tion performance compared to random task ordering. More recently, Bell & Lawrence (2022) investigated task-order optimization by examining Hamiltonian paths on a task dissimilarity graph (see Sec. 4 for details). They hypothe-
2

---

Optimal Task Order for Continual Learning of Multiple Tasks
sized that the shortest Hamiltonian path would be optimal but instead found that the longest Hamiltonian path signif- icantly outperformed both random task ordering and the shortest path in continual image classification tasks. Our work provides analytical insights into when and why this is the case. Lin et al. (2023) analyzed generalization error and task-order optimization in continual learning for linear regression. Our work advances this theoretical framework in several important ways. First, we introduce a latent struc- ture model for considering the effect of input similarity and reveal how tasks' relative positions-not just their absolute positions as in Lin et al. (2023)'s Equation 10-influence the model's final performance. We validate this theoretical finding through experiments on both synthetic data and im- age classification tasks. Furthermore, we extend beyond the synthetic task settings of Lin et al. (2023) by demonstrating these effects in general continual learning framework us- ing data-driven similarity estimation. Task-order effects on continual learning have also been analyzed in (Pentina et al., 2015; Evron et al., 2023; Singh et al., 2023).
We sample elements of (Aps Bu)e= from corre- lated Gaussian distribution. Denoting vector con- sists of the (i,j)-th elements of AI,.,, Ap by dij = [A1.85, A2.5. ... ApRs)" we sample aij E R from
aij N (0, N. C
(2)
where C is a P x P matrices that specify input corre- lation between tasks. Similarly, we sample the (i, j)th elements of Bp, biy = [B1.aj. B2.ij...... Bryy] IT. by bis (V N. Cox)). Note that here correlation is introduced across tasks in element- manner while keeping elements of each mixing matrix independent (i.e. (Ar,
Ō5H N, where ô5 represents the Kro- necker delta. Here we generate the model from the task similarity matrices (Cri, Coou] because previous work sug- gests the crucial impact of task similarity on continual learn- ing (Ramasesh et al.: Lee et al., 2021). In section 5, we consider the estimation of task similarity from datasets to ensure the applicability of our framework.
The linear teacher-student model used in this work is a widely adopted framework for analyzing the average prop- erties of neural networks by explicitly modeling the data generation process through a teacher model (Gardner & Derrida, 1989; Zdeborová & Krzakala, 2016; Bahri et al., 2020). Due to their analytical tractability, these models have offered deep insights into various aspects of statistical learn- ing problems, including generalization (Seung et al., 1992: Advani al., 2020), learning dynamics (Saad & Solla, 1995; Werfel al., 2003; Saxe al., 2014), and representation learning (Saxe et al., 2019; Tian et al., 2021). Many studies have also applied this framework to explore various aspects of continual learning (Asanuma et al., 2021; Lee et al., 2021; Evron et al., 2022; Goldfarb & Hand, 2023; Li et al., 2023; Lin et al., 2023; Goldfarb et al.; Hiratani; Mori et al.).
Let us consider the training of a linear network, y Wa: in this set of P tasks. We evaluate the performance of the network for the ju-th task using the mean squared error:
<n[[]= = (1 -1l"" = BB - W Aulls
(3)
where ()), represents expectation over latent (O, T). In task-incremental continual learning task (Van de Ven & Tolias, 2019), we are mainly concerned with minimizing the total error on all tasks after learning all tasks. Denoting the network parameter after learning of the last, P-th, task by Wp, the final error is defined by
P
1
Ef
Eu Wr].
(4)
N
3. Task-order Dependence
Below, we take the expectation over randomly generated mixing matrices (Au, B and derive the average final error é = (ef) (B, as function of the input and output correlation matrices Cr and Cout, Subsequently, we ana- lyze how the task order influences ě and how to optimize the order.
3.1. Model Setting
Let consider a sequence of P tasks, where the inputs E R= and the target output y" E RR, of the 4-th task is generated by
~ N(,T), a x= Aps, 3¹ = Bys..
3.2. Analysis of the Final Error Ef
Here, s E N. the latent factor that underlies both æ and y", is the identity matrix, and A,, NeNN, and Bμ E IR Ny N. are the mixing matrices that generate the input x and the target y² from the latent (Fig. 2a). Below we focus on N₂ >> N₂ regime. The introduction of this low-dimensional latent factor s is motivated by the presence of low-dimensional latent structures in many real-world datasets (Yu et al., 2017; Cohen et al., 2020).
We consider task incremental continual learning where P tasks are learned sequence one by one. Let us denote the weight after training on the (μ (μ-BB 1)-th (1-h)-th task as W Considering learning of the μ-th task from W = W us- ing gradient descent on task-specific loss cp. (n[W], the weight after training follows (see Appendix A.2)
Wp -1 1 (I U,U,)) + B,
(5)
3

---

Optimal Task Order for Continual Learning of Multiple Tasks
P=
P=
P=7
Total error (theory)
3.5
Ay
2.
8
y
20 24 30 A1
Total error (numerical)
Task B-C correlation Pgt c
0.2
0.4
0.6
0.8
Optimal
1.0
1.0
1.0
fask order
0.8
0.8
0.8
0.8
A B
0.6
0.6
0.6
- C
A
0.4
0.4
0.4
B A
0.2
0.2
0.2
0.2
A
C - B A
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.5
Task A-B correlation
Figure 2. a) Schematic of the seacher-student model. b) Comparison between the analytical and numerical evaluations of the error ef under various number tasks. Each point represents errors under randomly sampled task similarity matrices (C"m, Crout) (see Appendix C for implementation details). c) Optimal task order for three task learning. In the white regions, C is not positive-definie matrix, hence the tasks are not well-defined.
where U,, is defined by singular value decomposition (SVD) of A,. A, = U vT, and A is the pseudo-inverse of A.
learning via gradient descent, under various choices of the number of tasks P and task correlation matrices O and Cout (each point Fig. 2b represents the errors under one randomly sampled {Crh, C.out) } pair). This result indicates that our simple analytical expression robustly captures the performance of continual learning a linear teacher-student model under arbitrary task similarity and the number of tasks.
Applying it recursively while assuming that W initialized as zero matrix prior to the first task, we have
Wa Σ (B II (I (U-))))) p=++1
(6)
If N₂ >> No. pseudo-inverse At approximated by a scaled transpose yAT and U approximately follows N. U z AA,A7 with y= N₂ (see Appendix A.4). Thus, under Na N₂ < 1, we have
To explore how task order influences the continual learning performance, we next calculated the optimal task order of three tasks under various input correlation C using Eq. (Fig. 2c). Here, we the output correlation Cjv 1 for all task pairs (n, v) for simplicity, and parameterized the input correlation between tasks A, B, and C by
WP (B.A)) II
Under this approximation, there exists simple expression of the final error below (see Appendix A.3 for the proof). Theorem 3.1. As N. cally satisfies
PAB
PCA
N= - limit, the final error asymptoti-
C
PAB
PBC 1
(9)
PCA
PBC
& -=||(Cou11180 (T- (1- +T
(8)
where U is the strictly upper-triangle matrix generated from the input correlation matrix Cr (see eq. 34).
Here, tasks A, B, and are linear regression tasks with partial overlap in the input domain. РAB = 1, the input subspace for tasks and are the same, while they are independent if PAB O. Figure 2e revealed that the optimal task order depends
Importantly, the dependence on the upper-triangular com- ponents in Eq. 3.1 implics that ε not permutation- invariance, and thus depends on the tadk-
the combination of task similarity (PAB, PBC: PCA) in rich and complex manner. Some of the phase shifts represent trivial mirror symmetry (c.g., r = line), but many of them are non non-trivial. To further gain insights into this complex task-order dependence, below, we consider the linear perturbation limit.
3.3. Numerical Evaluation
To check this analytical result, in Fig. 2b, we compared ē estimated from Eq. with its numerical estimation through

---

Optimal Task Order for Continual Learning of Multiple Tasks
Periphery core
Gore - pariphery
Min paih
path
le
Task μ
ds
-05
G
1a
G
Task
e
f
Core-te-periphery Periphany-to-core
0.50
0.75
G
Relative error
Coefficient
0.0-
P=7
P=
Relative error
Min patl) 1.6-
Factor
1.50
0.55
1.2
1.35
0.25
Ghapp
-10-
Max path
-0.50
6.75
-15
0.50
-0.50
-022
0.00
0.25
0.50
0.75
1.00
-0.1
6.0 0.1 0.2
Correlation m
Average correlation m
Correlation m
Average correlation m
Figure 3. Linear perturbation analysis of task order dependence. a) Decomposition of the contribution of task similarity to the final error under P. 1, m 0.3. b) Schematic representations periphery-tee caanerr--to-oofe to-eore ordering and core-to-peripee toe-ttp--prppyyy ordering. Circles A, B. represent tasks and their spatial positions represent similarity between tasks. Here, lasks A-C are central whereas tasks D-I are periphery. under P = 7. Colors indicate the
e) Schematic of minimum and maximum paths on a dissimilarity graph. d) G as function indices such that the purple line on the top corresponds to Gi.2. while the yellow line the bottom corresponds to G e) Relative error of core-to- perriphrr and periphery-to-con error random seeds (see Appendix C for details). Coefficient limit. g) Relative error of min min-path and max max-path task orders under P 7. Here we took average over two task orders that follows the minimum pathway to estimate the error of min-path, The error of max-path was estimated in the same manner.
rules under various average task correlation m at P = 7. Error hars represent the standard
under 1 3, Dark gray line corresponds a at P + so
4. Task-order Optimization
Then, the final error is written below:
4.1. Linear Perturbation Theory
P
P
Ey E Crmut = EEy[m; + Σ Σ
Gpr + 0(6,))
Theorem 3.1 revealed a simple relationship between the task similarity and the final eгтor of continual learning, but it remains unclear how optimize the task order for continual learning. To gain insight into this question, next add small perturbation to the input similarity matrix and examine how the change in the similarity between various task pairs modifies the error. We parameterize the input correlation matrix by a combination a constant factor and a small perturbation.
-=Iv=tt
(11)
where & [m, Pol the error the absence of perturbation, and Gyv is a function of m. Par and P (see Eqs. 57 in Ap- pendix). At Po 1, Gaw has a following simple expression:
Gpv = G+ + G.uur
(12a)
yv
G+ = -m)++H-1 - (1-m) -
3-m (1 mm)-u-1
(12b)
2-m
Cinn per
(if ==
(10)
G = pv
m)
1-m
mP +
3-m
2-m
(1- P-
m + 8Ma (otherwise).
(12c)
Note that, P x P matrix G specifies the contribution of (H,, v)-th task similarity to the final error. The proof of the theorem is provided in Appendix B.2. Fig. 3a describes an example of Gav (here P 7 and m 0.3). In this case, G12 is positive while G17 is negative, meaning that if you increase the similarity between the first and the second tasks while keeping the rest the same, the total error é goes up, but if you increase the similarity between the first and the last tasks, the error instead decreases. To understand this task order dependence, we next analyze + and G separately.
Here, we set the constant factor m to be the same across all tasks for the analytical tractability of the matrix inversion, and perturbation 8M is constrained to the ones that keep Cr to correlation matrix. Similarly, target output correlation matrix to be Cout Po for all non -on-diaonnal components. In this setting, the error has the following decomposition.
restricted the
Theorem 4.1. Let us suppose that all elements of matrix 8M satisfies, Mp 8mm where 8m is a positive constant.
5

---

Optimal Task Order for Continual Learning of Multiple Tasks
4.2. Impact of Task Typicality
one another in the task order sequence should be dissimilar to each other, while distant pairs should be similar.
Let us first consider the contribution of G term. Denoting at
2-m (1 m)", G is rewritten as
Given a set of tasks, let us define a task dissimilarity graph by setting each task as a node and dissimilarity between two tasks as the weight of the edge between corresponding nodes (Fig. 4c). Then, a task order that learns each task only once forms Hamiltomian path on the graph, a path that visits all nodes once but only once. We can then construct heuristic solution for minimizing oc7 by selecting task order that yields the longest Hamiltonian path. When tasks have the same similarity with each other in terms of Cout, their similarity depends solely on C allowing us to define dissimilarity as dur = 1 - C Thus, the total length of the Hamiltonian path induced by a given task order follows DH = Epmi drwtt.. Consequently, dej rewritten as
3 3-m
juu
G
(2-m)(1-m) m) ((1 -m) at) ((1-m) a")
3-m
aa" + (1 m) P-1 p
(13)
If 1 > m O. (2 (2-m)(1-m) > and (1 - m)y (1 m)² > at for μ = P. P. Therefore, G is monotonically decreasing function of both μ and v under 1 > m > 0 (Fig. 3d). This means that, to minimize the error contributed from G ju dej- = ΣnE, Juu JttoMM the tasks should be ordered in a way that the residual similarity 5MM takes a small (preferably negative) value for early task pairs and a large value for later task pairs. In other words, earlier pairs should be relatively dissimilar to each other, while later pairs should be more similar.
3 3-m
2
8c7 -a ((1 P-1P-1D
p 1 (1 m) p- 2=2
P-d dppatd + const.
One heuristic way to achieve this task order is to put the most atypical task at the beginning and the most typical one at the end. Denoting the relative typicality of the task by St, = Eurz 6Mu if we arrange tasks as St, S 6tz S ... ...S ptp, on average, earlier pairs are dissimilar to each other while the latter ones similar. Below, we denote this ordering as periphery-to-core rule, as less representative periphery tasks are learned first and more central core tasks are learned later under this principle (Fig. 4b). Under a randomly generated input correlation matrix Cr²", periphery- to-core task order robustly outperformed both random and core-to-periphery order, when the average correlation is large positive value (red vs black and blue line in Fig. 3e). This was not the case when the average correlation is small positive value potentially due to contribution from G- factor. Note that, a similar rule was derived by Lin et al. (2023) based on their analysis of linear regression model, where they proved that when there is one outlier task, the outlier task should be learned in the first half of the task sequence.
(15)
Because -a is non-negative, small task dissimilarity du (i.e., large task correlation Ci generally helps minimiz- ing the error. Moreover, we have 0 S (1 - -m)²= < (1 - 1-)mm-d for d = 2,3,... indicating DH term has the smallest impact on the error. Therefore. by choosing the largest dy for DH. we can make 8e, small on average. We observed this trend robustly even when we sampled {Cin, Couut randomly (Fig. 3g). Our work thus provides theoretical insights on why the maximum Hamiltonian path provides a preferable task order, strengthening previous em- pirical finding (Bell & Lawrence, 2022). We call this rule as max-path -at-hath rule below.
4.4. Application to Tasks Having Simple Graph Structures
The analyses above elucidated two principles underlying task order optimization. To illustrate these principles, we next examine task order optimization for set of tasks with simple task similarity structure.
4.3. Impact of Hamiltonian Path Length
Let us next focus on G term that governs the contribution of the relative distance between tasks in the task sequence. The error originating from this term is written
Figure 4a depicts the total error estimated using Eq. 8 in a continual learning scenario involving five tasks with a chain- like similarity. We configure the input correlation matrix Oi such that tasks A and are directly correlated, while and are correlated only indirectly through B. Specifically, denoting the similarity between neighboring tasks on the task dissimilarity graph as we set Can = Cin o, Cir = a², and so on (see Appendix C). Here, tasks ex- hibit significant overlap when a S 1, while tasks become independent in the limit a + 0 (x-axis of Figure 4a). We set Coutt to one for all task pairs. Each line in Figure 4a
Jeg = a
m)
P
8M Mupdd
(14)
d=1
is a coefficient. a" is negative if > m >0 0 and iss sufficiently large (Fig. 4f). Thus, to minimize the error _ the tasks should be arrange in a way that 8M is small for small d, while d is large for large d. In other words, tasks following
where a = -1-
mP I-m
3-m 2-m
6

---

Optimal Task Order for Continual Learning of Multiple Tasks
Chain graph structure
b
Ring graph sirusture
Chain (numerical)
Total errr (teoryy)
Total (theory)
0.
Total arror
3.
60
10
24
06
04
Task correlation
Task correlation 8
Training time
O
(0000)
Tree graph structure
Tree graph (leeves only)
Trea (numerical)
Total error (Sheory)
Total erar (theary)
1.6
Tea sror
Task correlation
Task correlation
Training time
Figure 4. Optimal task orders for tasks with simple graph-like like similarity structures. a,b) Total error ef under all task orders when the similarity structure of five tasks follows chain (a) and ring (b) structures. Each gray line represents one of 120 (=5!) task order, while red, orange, and blue lines highlight three representative task orders depicted on the right. Thick black line is the average emror over ordering. <) Numerically--xtimmaee learning dynamics of the network when tasks have a chain-graph structure. Red and blue lines represents A-B CD +E and A E C +D >B orders depicted in the insets. Black line is the average learning trajectory under random task ordering. d,e) The same as panels and b but for tasks with similarity matrices having tree (d), and tree-lcaves similarity structure (c), respectively. The same as panel c, but for tasks having pre-graph-like similarity structure.
represents the error under a specific task order. For example, the A B----C---E task order, depicted by the red line, consistently performed among the worst, regardless of the similarity a between neighboring tasks. Surprisingly, sev- eral task orders robustly outperformed the others, such as A --C-EED--- (orange line) and A AB (blue line). These task orders align with the two principles de- scribed earlier. First, the periphery-to-cov that the initial task should either be A or E, as these tasks are the least typical.' Second, the max-path rule indicates that subsequent tasks should be as dissimilar as possible. For instance, if the first task is A, selecting E as the second task, as in the blue line, maximizes the distance. Notably, there was approximately seven-fold difference in performance between the best and worst task orders, underscoring the critical importance of task order in continual learning.
Numerical simulations validated the analytical results (Figs. 4c and 4f) and further revealed intricate learning dynamics. In Figure 4f, the red task order initially outper- formed the black line representing the average performance, while the blue task order performed worse. However, this trend reversed around the fourth task. These findings indi- cate that the optimal task order is often non-trivial. and a greedy approach optimizing task-by-task error may lead to suboptimal performance.
rule suggests
5. Application to Image Classification Tasks
Our analytical investigation in the linear teacher-student setting highlighted two principles for task order optimiza- tion: the periphery-to-core rule and the max-path rule. To evaluate the potential applicability of these principles to more general settings, we next explore continual learning of image classification tasks.
We observed analogous trends when applying the same anal- ysis tasks with ring, tree, and leaves structures (Figs. 4b, 4d, and 4e, respectively). For tasks with tree-like similarity structure, as shown in Figure 4d, the eиror was minimized when tasks corresponding to leaf nodes were learned first, followed by tasks associated with root nodes (the orange and blue trees in Fig. 4d). This result aligns with the periphery- to-core rule. When only the leaf nodes were considered as tasks, as illustrated in Figure 4e, the optimal task order exhibited a complex pattern of hopping across tasks (blue tree in Fig. 4c), consistent with the max-path rule.
5.1. Empirical Estimation of Task Similarity
To apply these principles, it is necessary to first measure the similarity between tasks. Here, we estimate the similarity between tasks A and B by measuring the zero-shot transfer performance between them (Fig. 8). Specifically, we train network for task A, obtaining the learned weights WA. A. We then measure the error of this trained network on task B, denoted as e [WA]. Since the transfer performance from task A to B generally differs from that of B to A we take
'Due mirror symmetry, the E-AA-CBB +D order exhibits cquivalent performance to A -E CBD
7

---

Optimal Task Order for Continual Learning of Multiple Tasks
a
Fashion MNIST
C
CIFAR-10
CIFAR-100
Periphery-lo-cora Core-lo-eriphery
Periphery-lo care
0.8
Periphery-lo-cort Average
Periphery-to-coe Average
1.0
Performance
5.8
Average
Core- to periphesy
Performance
Core- to periphery
Performance
Core-lo-periphery
8.7-
Ratio
0.55
86
2.5
0.5
P 7
0.5
0.5
2.5
0.6
Ps 5
P 3
0.3
0.4
-0.1
0.1 Average similarity
0.0
0.2
-0.1
0.0
0.3
0.3 04
Average similarity
Average similarity
Average similarity
b
h
Max path Min path
Max path
Max path
Max path
1.0
03.
Average
Average
Average
Performance
Min path
Perforance
Min path
Performance
0.7
Min
0.9
Ratio
0.5
0.5
0.6
0.7
P
0.5
P= 5
P 3
0.4
0.4 -0.1
0.0
0.1 0.2
0.5
-62
-0.1
0.0
0.1
0.2
0.3 0.4
Average similarity
Average similarity
Average similarity
Average similarity
Figure 5. Task order preference in continuous image classification tasks, a-f) Continual learning performance, defined as the average test accuracy across all the tasks after learning. under various task orders. Panels (a, c, e) compare the periphery-to-on rule against the core-to-perin core-to-periphery rule, whereas panels (b, d. f) compare the max-pat max-path rule with the min-path rule. g, h) The ratio of task sets where the periphery-to-core rule outperforms the cone-to-periphery rule (g), and where the max-path rule outperforms the min-path rule (h), under CIFAR-100. Different colors represent results for different numbers of tasks (P = 3, 3,5,77 7). See Appendix C.3 for details.
the mean of both directions and define the similarity PAB:
tional layers and one dense layer, to explore robustness against the model architecture.
(v1
EB [W]] CB,ay|WAl [WA] +
ea(We] CA.sf [WB]
PAB = 2
(16)
We found that the final performance was modulated by the estimated average similarity among tasks, p P(P-1j Ensr Pau, we thus plotted the performance of each task-order rule as a function of the average similarity (Fig, 5). In all three settings, we found that the periphery-to- co-oee rule robustly outperforms the torr-to-perpphey rule and average performance over random ordering (Fig. 5a,c,e), Similarly, the max-path rule outperformed both the min- path rule and the random ordering (Fig. 5b,d,e; see also Bell & Lawrence (2022)). Moreover, we observed consistent results under a continual learning of a multi-class classifica- tion (Fig.7a and b). The periphery-to-core rule outperformed the max-path rule on average, but the difference was small (red lines in Fig. 5 top bottom). When we increased the number of binary classification tasks from 3 to 7 using CIFAR- 100, the performance advantage periphery-to-con over core-to-ped to-periphery increased (Fig. 5g) as expected from the linear model (Fig. 6e). This was not evident for max- path and min-path rules potentially because the difference was already high under P 3 (Fig. 5h).
Here, 6B.sf [Wa] represents the error on task B with label shuffling, which serves the chance-level error. The square root is taken because the error scales with the squared value of task correlation in our linear model (see Appendix C.4).
Although this method requires training the network on all P tasks, the computational complexity of training is ((P), which is significantly smaller than the naive task order opti- mization that requires computational cost of O(P). Fur- thermore, this method only requires the inputs and outputs of the trained network, making it applicable even in situa- tions where the model's internal details are inaccessible.
5.2. Numerical Results
We estimated the performance of the periphery-to-com rule and max-path rule in ias-iinrrementl continual learning using Fashion-MNIST (Xiao et al., 2017), CI, and CIFAR-100 dataset (Krizhevsky et al., 2009) (see Appendix C for the details). For the Mishion-MNISTT and CIFAR-10 datasets, we randomly generated five binary image classi- fication tasks by dividing 10 labels into 5 pairs without re- placement. In the case of CIFAR-100, we selected 10 labels out of 100 labels randomly and generated 5 binary classi- fications. For Fashion- MMNIST, we trained a multi-layered perceptron with two hidden layers, while for CIFAR- LIFAR-10/100 10/100, we used convolutional neural network with two convolu-
We also investigated the inference of task similarity and ordering from a small subset of training data. This exten- sion is crucial, as it demonstrates the practical relevance of our theory to real-world machine learning settings where full access to all training data upfront is often unrealistic (in contrast, when complete data is available, naive multi- task learning may suffice). When reduced the number of training samples used to estimate task similarity across tasks
8

---

Optimal Task Order for Continual Learning of Multiple Tasks
in CIFAR-10, the relative advantage of both the periphery- to-core rule and the max-path rule over random task order remained robust. Even when only 1 % of the training data was used for estimating task similarity, we observed a per- formance gain comparable to that in the full data scenario (Panels and vs. g and h in Fig. 9). However, when the amount of data was reduced to 0. % (approximately 10 sam- ples per task), the performance gain became non- sinniiiiiiin... We observed similar trends with both the Fashion- -NNIST and CIFAR-100 100 datasets, although the results for CIFAR- 100 were less robust, particularly in the negative similarity regime (Fig. 10). These results suggest the robustness of the task order optimization principles found in our simple analysis.
Impact Statement
Task order optimization for continual learning of multi- ple tasks may potentially contribute beyond the field of machine learning from school curriculum design (Rafferty et al., 2016; Zhu et al., 2018) to animal training protocol in neuroscience experiments (Krueger & Dayan, 2009). Nev- ertheless, due to theoretical nature of this work, there are no specific societal consequence that we feel must be high- lighted here.
References
Advani, M. S., Saxe, A. M., and Sompolinsky, H. High- dimensional dynamies of generalization error in neural networks. Neural Networks, 132:428-444 2020.
6. Discussion
Asanuma, H., Takagi, S., Nagano, Y., Yoshida, Y., Igarashi, Y., and Okada, M. Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks. Journal of the Physical Society of Japan, (00100)1040011 2021.
In this work, we derived a simple analytical expression to explain how task similarity and ordering influence continual learning performance in a linear model with latent struc- ture. Based on this result, we proposed two principles for rule and the
task order optimization: the periphery-tos max-path rule, the latter of which was predicted by Bell & Lawrence (2022). We validated these principles in task- incremental continual image classification tasks using both multi-layer perceptrons and convolutional neural networks. Thus, this work proposes basic principles for task order op- timization in the context of continual learning for multiple tasks.
Bahri, Y., Kadmon, J., Pennington, J., Schoenholz, S. S., Sohl-Dickstein, J., and Ganguli, S. Statistical mechanies of deep learning. Annual Review of Condensed Matter Physics, 11:501-528, 2020.
Bell, S. J. and Lawrence, N. D. The effect of task ordering in continual learning. arXiv preprint arXiv:2205.133: 13323, 2022.
Limitations
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Our theoretical results were derived in a linear model under the assumption of random task generation, which limits their direct applicability. However, we numerically confirmed that the proposed ordering rules hold in both convolutional neural networks and multi-layer perceptrons trained for con- tinual image recognition tasks. Future work should further evaluate these rules in domains closer to real-world appli- cations, including dee robotics, and language models. Additionally, in this work, we restricted the model setting to scenarios where each task is learned only once and trained to convergence. The first assumption can be readily re- laxed as long the total number of tasks remains small (see Appendix A.4). Relaxing the second assumption is important direction for future work.
Cohen, U., Chung, S., Lee, D. D., and Sompolinsky, H. Separability and geometry of object manifolds in deep neural networks. Nature communications, 1 11(1):746, 2020.
Elman, J. L. Learning and development in neural networks:
The importance of starting small. Cognition, 48(1):71-99, 1993.
Evron, I., Moroshko, F.,, Ward, R., Srebro, N., and Soudry, D. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory. pp- 4028-4077 PMLR, 2022.
Acknowledgements
This work was partially supported by McDonnell Center for Systems Neuroscience.
Evron, I., Moroshko, E., Buzaglo, G., Khriesh, M., Mar- jich, B., Srebro, N., and Soudry, D. Continual learning in linear classification on separable data. In Interna- tional Conference on Machine Learning, pp. 9440-9484, PMLR, 2023.
9

---

Optimal Task Order for Continual Learning of Multiple Tasks
French, R. M. Using semi-distributed representations to overcome catastrophic forgetting in connectionist net-
Ke, Z., Liu, B., Xiong, W., Celikyilmaz, A., and Li, H. Sub-network discovery and soft-masking for continual learning of mixed tasks. In The 2023 Conference Empirical Methods in Natural Language Processing.
works. In Proceedings of the 13th annual cognitive sci-
ence society conference, volume 1, pp. 173-178, 1991.
French, R. M. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4): 1999.
Ke. Z., Liu, B., and Huang, X. Continual learning of mixed sequence of similar and dissimilar tasks. Advances in neural information processing systems, 33: 18:18843311804, 2020.
Gardner, E. and Derrida, B. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 22(12):1983, 1989.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Gaabska-Barwinsker A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of national academy of sciences, 1 114(13):3521-3526, 2017.
Goldfarb. D. and Hand, P. Analysis of catastrophic forget- ting for random orthogonal transformation tasks in the overparameterized regime. In International Conference on Artificial Intelligence and Statistics, pp. 2975-2993. PMLR. 2023.
Kontogianni, T., Yue, Y., Tang, S., and Schindler, K. con- tinual learning ready for real-world challenges? CoRR, 2024.
Goldfarb, D., Evron, I. Weinberger, N., Soudry, D., and HAnd, P. The joint effect of task similarity and overpa- rameterization on catastrophic forgetting-as an analytical model. The Twelfth International Conference on Learn- ing Representations.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Krueger, K. A. and Dayan, P. Flexible shaping: How learn- ing in small steps helps. Cognition, 110(3):380-33 2009.
Guo, M., Haque, A., Huang, D.-A., Yeung. S., and Fei-Fei, L. Dynamic task prioritization for multitask learning. In Proceedings of the European conference computer vision (ECCV), pp. 270-287, 2018.
Kumari, P., Chauhan, J., Bozorgpour, A., Azad, R., and Merhof, D. Continual learning in medical imaging analy- sis: A comprehensive review of recent advancements and future prospects. CoRR, 2023.
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. Embrac- ing change: Continual learning in deep neural networks. Trends in cognitive sciences, (12):11222 1040, 2020.
Lad, A., Ghani, R., Yang, Y., and Kisiel, B. Toward optimal ordering of prediction tasks. In Proceedings of the 2009 SIAM International Conference on Data Mining, pp. 884- 893. SIAM, 2009.
Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Zee, M. Flax: A neural network library and ecosystem for JAX, 2024. URL http:// github. com/ google/Elax.
Helias, M. and Dahmen, D. Statistical field theory for neural networks, volume 970. Springer, 2020.
Lee, S., Goldt, S., and Saxe, A. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pp. 6109- 6119. PMLR, 2021.
Hiratani, N. Disentangling and mitigating the impact of task similarity for continual learning. In The Thirty-eighth Annual Conference Neural Information Processing Systems.
Lesort, T., Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., and Diaz-Rodrígun N. Continual learning for roboties: Definition, framework, learning strategies, opportunities and challenges. Information fusion, 58:52-68. 68, 2020.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbi, J., Clark, A., et al. Training compute- optimal large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, pp. 30016-30030, 2022.
Li, C., Huang, Z., Zou, W., and Huang. H. Statistical mechanics of continual learning: Variational principle and mean-field potential. Physical Review E, 108(1): 014309, 2023.
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., and Levine, S. How to train your robot with deep reinforce- ment learning: lessons we have learned. The International Journal of Roboties Research, 40(4-5):698-72) 2021.
Lin, S., Yang, L., Fan, D., and Zhang, J. Beyond not- forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35:16165-1617,, 2022.
10

---

Optimal Task Order for Continual Learning of Multiple Tasks
Lin, S., Ju, P., Liang, Y., and Shroff, N. Theory on forget- ting and generalization of continual learning. In Inter- national Conference on Machine Learning, pp. 21078- 21100. PMLR, 2023.
Saxe, A. M., McClelland, L.,, and Ganguli, S. A mathe- matical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sci- 2019.
ences. 116(23)
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint ar www.::330..0874. 2023.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp. 4548 4557. PMLR, 2018.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165, Elsevier, 1989.
Seung, H. S., Sompolinsky, H., and Tishby, N. Statistical mechanics of learning from examples. Physical review A, 45(8):6056, 1992.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances in neural informa- tion processing systems, 30, 2017.
Mori, F., Mannelli, S. S., and Mignacco, F. Optimal pro- tocols for continual learning via statistical physics and control theory. NeurlPS 2024 Workshop on Mathemat- ics of Modern Machine Learning.
Singh, P., Li, Y., Sikarwar, A., Lei, S. W., Gao, D., Talbot, M. B., Sun, Y., Shou, M. Z., Kreiman, G., and Zhang, M. Learning learn: How continuously teach humans and machines. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11708-11719, 2023,
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P. Currieulum learning for reinforce- ment learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-555 2020.
Tian, Y., Chen, X., and Ganguli, S. Understanding self- supervised learning dynamics without contrastive pairs. International Conference on Machine Learning, pp. 10268-10278 PMLR, 2021.
Peng, L.., Giampouras, P., and Vidal, R. The ideal continual learner: An agent that never forgets. In International Con- ference Machine Learning, pp. 27585-27505 27610. PMLR, 2023.
Van de Ven, M. and Tolias, A. S. Three scenarios for continual learning. arXiv preprint arXiv:1904.0777: 2019.
Pentina, A., Sharmanska, V., and Lampert, C. H. Curriculum learning of multiple tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5492-5500, 2015.
Verwimp, E., Yang, K., Parisot, S., Hong, L., McDonagh, S., Pérez-Pellitero, E., De Lange, M., and Tuytelaars, T. Clad: A realistic continual learning benchmark for autonomous driving. Neural Networks, 161:659-669 2023.
Rafferty, A. N., Brunskill, E., Griffiths, T. and Shafto, P. Faster teaching via pomdp planning. Cognitive science, 40(6))1290-1332. 2016.
Werfel, J., Xie, X., and Seung, H. Learning curves for stochastic gradient descent in linear feedforward net- works. Advances in neural information processing sys- tems, 16, 2003.
Ramasesh, V. V., Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Rep- resentations.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mmnn a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: / 1708.07777, 2017.
Robins, A. Catastrophic forgetting, rehearsal and pseudore- hearsal. Connection Science, 7(2):123-146, 1995.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Yu, X., Liu, T., Wang. X., and Tao, D. On compressing deep models by low rank and sparse decomposition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7370-7379, 2017.
Saad, D. and Solla, S. A. On-line learning in soft committee machines. Physical Review E, 52(4))4225, 1995.
Zdeborová, L. and Krzakala, F. Statistical physics of infer- ence: Thresholds and algorithms. Advances in Physics, 65(5):453-552 2016.
Saxe, A., McClelland, J., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings the International Conference on Learning Represenatations, 2014.
Zhu, X., Singla, A., Zilles, S., and Rafferty, A. N. An overview of machine teaching. arXiv preprint AXXi:/80110092111 2018

---

Optimal Task Order for Continual Learning of Multiple Tasks
A. Analysis of the Impact of Task Similarity on Continual Learning
A.1. Model Setting
Below, we amalyze task-order dependence of continual learning using linear teacher-student models with a latent factor. In teacher-student models, the generative model of the task parameterized explicitly by the teacher model, making the learning dynamics and the performance analytically tractable (Gardner & Derrida, 1989; Saad & Solla, 1995; Zdeborová & Krzakala, 2016). Here, the generative model for input RR.,,, and the target output y R.Vv is constructed as
N (0, I), A48, y¹ = Bys,
(17)
where 8 E R-N is the latent variable that underlies æ and y / is the identity matrix, and A, E RRNxNN, and By E NRN, are mixing matrices for the input and the target output at task μt = 1,., respectively.
We generate matrices (An: B randomly but with task-to-task correlation. We specify the element- wise correlation among input generation matrices { P y= by a P X P correlation matrix C and specify the correlation among the target output generation matrices {By] by another P x P correlation matrix Cou,, C and Cost are constrained to be correlation matrices, but arbitrary otherwise. For all 1 < i S N₂ and 1 j < No, we generate (i, elements of matrices A1,... Ap by jointly sampling them from a Gaussian distribution with mean zero and covariance 1 C N,
A : A
N 0, NN Co
(18)
Similarly, we generate (i, elements of matrices BI....,Bp by (B.)...., BBY ~ ~N(O, Cam)). Note that, under N. this construction, two different elements in a matrix A,, are independent with each other, but the same element matrices for two different tasks are correlated with each other. Although the random task generation assumption limits direct applicability of our theory, it enables to obtain insights into how task similarity influences the overall performance and optimal task order. Moreover, our analytical results up to Eq. 28 hold for arbitrary mixing matrices {A,, By they don't 't assume the expectation over {Ay, B,
The student network that learns the task is specified to be linear network:
Wæ,
(19)
where W N X is the trainable weight. The mean mean-squared error between the output of the student network y and the target for the j-th task y" is given by
Sa W] N, (1u- N
1 B W WA
(20)
The second equality follows from the Gaussianity of We consider task-incremental continual learning (Van de Ven & Tolias, 2019) where the network is trained for task μ 1.... P in sequence. During the training for the f--th task, weight W is updated by gradient descent on error Ep:
dcg W
2n N₂
W W
W
(B, - (B.-
WA,)
(21)
OW
We denote the weight after training on task as WT, The total error on all tasks at the end of all P task learning becomes:
P LBB 118.-WPA.l Wp
Ef Σ cu [W N2 n=1
(22)
To uncover how similarity between tasks influences the final error, in this work, we focus on the average performance over randomly generated mixing matrices (Ap: B under fixed pair of task correlation matrices Cr Coot, We define the average of the final error ey over generative models {Ap, B by
E (E)) Below, we first derive the analytical expression of Wp then estimate the total final error ey-
(23)
12

---

Optimal Task Order for Continual Learning of Multiple Tasks
A.2. The Weight after Continual Learning of P Tasks
Considering the gradient flow limit of learning dynamics, the weight update (Eq. 21) rewritten as dW
dt = (B, - WA
(24)
Let us denote singular value decomposition (SVD) of A, by Av UUADV where Up E R...N. No is non-negative diagonal matrix. Then, any point during learning, there exists a matrix Q(1) E RNNxN.o such that (() is written as W()) WN-1 Q(1)7
XN, and V, E RNN,N.
semi -rthnoratooolall matrices (i.e., U VTV I) and A2 E R RN..N.
because weight change during learning of the f--hh task is constrained to the space spanned by UT- Thus, at the convergence of learning, dW dr
0, we have
BB - p- 22077A,) 4 =0
(25)
Solving this equation with respect to Qu. we get Q₁ = B Ww-1Un- Therefore, the weight after training on the u-th task becomes
Wp Wp UnUU) + A
(26)
where A is the pseudo-inved inverse of A, (A) rewritten as
By applying this result iteratively from zero initialization, W
#
W (B.A II (1-)))))
(27)
1=1
p=v+1
where
is the identity matrix if u = v, otherwise,
II (1 - U,-% -1 -- p=
... ........--- (I U,U,)
(28)
To further investigate how task similarity impacts continual learning performance, below we focus on the large N2 regime, N, and analyze the learning behavior N_ + limit. This assumption of the presence of din-dimeniional latent factor consistent with many real-world datasets (Yu al., 2017; Cohen al., 2020). If Aμ is very-tall random matrix
(i.e. No >> N.). pscudo-inverse A is approximated by scaled transpose 7 and UNUT approximately follows UNUT Rs A where 7 =
Na Nr (see Appendix A.4). Thus, we have
W = (B.AD II (I-,7),)
(29)
p=++1
Using the approximation from Eq. 29, the error on the 4-thh task after training on w-th task, e, (,,[Wr),in is
e [W. B W.Au
B₁ - Σ (B)))) II (1 -A.
(30)
a=p+1
A.3. Proof of Theorem 3.1
N. Substituting Wp with Eq. 29, at N.
limit, Ey is rewritten as
P Σ
P
E N
1
B₁ -
B
B
A
II (1 -1-7Ag
((-7A,A))A
1=1
d=p+1
1
27
[
Bd Bp
B,
II (I A,,
N
Ng
p=1 p=1
g=pt1
P
P
2
B B
II (1-
(31)
0=p+1
13

---

Optimal Task Order for Continual Learning of Multiple Tasks
Taking expectation over (Au, Bp Bpla the first term is ( (IBWIle) Ny. The second term is rewritten as
P
P II (1 74, 1
tr
BY
B 0=p+1
N
P
ΣΣ
Cooe
(ir
[A]] 7Ap-1 (:I-7A44)A))))
NA
μ=l
p=1
P
1
Σ
Couut
M -7
Σ
A,
Σ
Σ
tr Ao, A7 Ao,
A Apl
N
dy=pt1
= d==a
P
P
P-p
Crout
(->)
Σ PEOSS
(r[AAAA,
A'" An))
A
(32)
N
k=
In the first line, we took expectation over (Bp}, which yields N, (ur B, N./N, N,C N, N, tr [M = N₂ tr tr[M] for arbitrary matrix M. In the third line, we rearranged the terms inside the trace based on dependence. The summation Epcay So <P in the last line is summation over set of indices condition. Under N₂ > N., the expectation term in the equation above follows (see Appendix A.4)
C
.... o% that satisfy p < < <00 <.. < ok P
(y"tr [A] A7, A. Anl) No (cro, Cén Can m.. + 0
N₂ No
(33)
Moreover, if we define an upper-triangle matrix Cin PXP by
Cin (if / <)
(34)
0
(otherwise)
we have
Σ
(tr [A7 AT,
Aos AT Apl)
Σ
Opor Oin oo.og...Con
+0 No
pcan p.ass..caas
pcoo CONP
= ((cin.))& cim]]
+ +0
No No
(35)
The last line follows because the upper triangle matrix Cin,U satisfies
P
P Σ
P Σ
Σ
Σ
Gial
Cin Cón.U Con.y
Con.U oc-10s ozf
Σ
Σ
Crén.U Cin.
Cin Cin Cin
O2-10s Copp
02
ai ppppl 02 +1
aumas 1+1
Σ
Ci Oin
Oioy ... 02-101
Cin
Oin
(36)
Moreover, because [( for any satisfies p P- k: + 1, we have
(Cin.li& Cin)
= Σ
(Cin.)) crin)
(37)
k=0
14

---

Optimal Task Order for Continual Learning of Multiple Tasks
Therefore, taking
limit, it follows that
P
P
M
Coou
Σ
(-))*
Σ
(1 Ao, Ab, ... AgeA Apl)
N
pp
R=0
pcor S..
(-1)+ [(Cinee) cin)
Cout
pp
Coor
(I c -1 Crn
PM
Cout (I + ( -1 Onn) (
(38)
In the third line, we used
(I + C
Σ(1) (Cimu() II
(-1) ( (Crn..))P+1 P+1
(39)
We can evaluate the third term of Eq. an analogous manner:
P
P
A
T
II (1 -
N
o=p+1
72
Σ
ΣΣΣN
BT
P II (I- (U-7A,A7)A, p=p+1
A3 II (1-7AAA43))) o'=p'41
N
N.
ΣΣΣ
Coutt (tr
-1 144
N
(40)
The term inside the trace can be expanded as
(tr (1 74p41 1.+t))... - 7Ap A,, -I (14
P P-pP-p
Σ
E
(-7)atk' (or Ao, Ao, AT, A A AT Aoy Ap.....A A.. (41)
F=1 pca S.. Co.
and the expectation over {An} follows (Appendix A. 4)
nkk' (tr [2 A% Ao, Ao, Ar, ...,A
A,, AT A A7 Ao; AT. Aot Ar, Ap
N2
(Cin Crnn
-o .......-Co Cnn Cin Cooy ...C C.p + O ( ):
N,
(42)
Nº > 0 limit, the squared term evaluated as Therefore, at No
P
P
2
B T
II
(T -
V
#=1
p=tt1
P-p
P-d
Cout
Σ (-1)
Σ
Cin
E
Σ
Ce ...
PSoIs...OEEP
k==
P
P
Σ
[[T + C/n (rin)
-
pu [(T
y [
pp"
(I Cii.tii--C (
(43)
15

---

Optimal Task Order for Continual Learning of Multiple Tasks
Noticing that the first term of Eq. 31 is rewritten as
1 N₂
Bull( 12)) =P- = tr(Cout,,
Bu
=t=rr[Coul]
(44)
at N, + limit, the final error Ey is written as N,
Ey tr(Crout] - 2tr (tr(Crout(I + ( [Couu[] I + C (1I Con
= (Coue) 1/2 ) (In
(45)
Thus, we obtained the equality in Theorem 3.1. Note that because Cout is correlation matrix, there exists a matrix (Coue)1/2 1/2 such that (Coot)1l22 (Coue) Cout, Because 7 + Cin,U + (Cim.U), .eg is also written as
Ef (Cout) 1/2 (I Cin.U)-- ( F.
(46)
Note that, if C = I, the error is zero. This is consistent with previous results showing that in the absence of overlap between tasks, continual learning doesn't t suffer from forgetting (Ramasesh et al.; Lee et al., 2021; Peng et al., 2023). Additionally, Eq. 33 requires P < N, (see Appendix A).4 below), thus the obtained expression doesn't 't hold when the number tasks is comparable to the network size.
N₂
A.4. Expectation over Random Correlated Matrices {Aμ}
We first show that A + 7AA and U,UT + 7A,A at N. No + O, where
= N, N. and Un is defined by SVD of A,,.
Aμ UnAaV VT If Ap J. then we have A UNA 7 UUU and and
UJ v (U,V =7 (+U = AT
= Vn
(47)
Thus, is sufficient to show that A2 + +x 1 at No sampled A, obey
N, 0. The mean and variance of N₂ x N, matrix A A, over randomly
(T)A)) N - = (1+11").
N,
(48)
where 1 is a all-one vector and 3 represents Hadamard product, indicating that the standard deviation of ATA,, N₂ scaling with respect to the mean. Thus, 74 + at
A, shows
Na N, + 0, implying Aμ. + + at
No N₂ + 0.
Regarding expectation over A in Eq. 33, expanding the equation up the next to the leading order, we have
(ur A1 Aaj A.. 1T Anl)
Σ
Σ
A A7 AP
A
A it- in
Ali
At AH
Aado lah 11
1132
1222
fu Jo
k+1
Σ
Σ
Carog- Cinn pa;
Onn
Nr
k+1
Σ
1 N.
Onn Cojog...C 20 Com Crdn Gin
1o Crin + ...)
N2
N: N.
Crn Chn Cin
N₂
N₂
Σ
pon Conag.... ... 1811 Caja agl
Crin Ciin
Cón Cin
N₂ O
N,
N,
=> N₂ (Cin, Oni C..m....Cin dap
+
N.
(49)
No
16

---

Let us consider the case when the output similarity is the same across all the tasks for simplicity. Then, Cout is written as Optimal Task Order for Continual Learning of Multiple Tasks
Jo..h...... in the third line is the Kronecker delta function that returns 1 if jo The third line follows from Isserlis' theorem, which states that the expectation over multivariate normal variables can be decomposed into summation over all pair-wise partitions (Helias & Dahmen, 2020). In the equation above, the partition that pairs neighboring matrices takes (NN*+1) value, while all other partitions yield ((N)) value at most because of indices mismatch. Note that, the number of second order terms depends on P, as suggested by the summation over I in the third line. Thus, we expect that our theory hold only when P satisfies P < Nz N,
... jks otherwise returns O.
From a parallel argument with the one above, the expectation in Eq. 42 is evaluated as
(ar
(4,An, Ao, Ab, ... Ar A A Ao,
A Aoo....A; Abi A2,Ar)
K+N
k+
N2
N₂ (Cpn,
Con
Conor...oon Cio,
...aaa Ca. +0 0
No
N₂
poi
N2
N₂
(Cpos par
Ope
in
Cin o2os d'pp'
+
No
(50)
B. Analysis of the Impact of Task Order on Continual Learning
B.1. Linear Perturbation Analysis of the Order Dependence
To further investigate the -rder-depen of the final error ey. we aim to decompose the error into interpretable features of task similarity matrix. To this end, we further constrain the input similarity matrix C to
(if i=
C
(51)
m + 8Mg (otherwise),
where m is constant satisfying - < m < 1. Here, 8M is a small element-wise perturbation added in such a way that Ca is a correlation matrix. We define an upper-triangular matrix that consists of the constant component as M. (i- element of M takes My = m if > i, but Me,
otherwise. This constant M assumption enables us to evaluate the effect of inverse matrix term in Ey analytically owing to the following lemma:
Lemma B.I. For any m satisfying -1 < m < 1, an upper-triangle matrix M satisfies
[(T + -)
1 = 6.g mlj > - (1-m11---
(52)
where [ is the indicator function that returns 1 if X is true, but returns 0 otherwise.
Proof.
P
[[T + M) + M)
Σ
(5%h + m[k x)+) (ōej mlj > 1|+(1 m))-k-1)
k=1
j-1
= 5i + > i|+ -m(1 m---t-1 + m m² Σ (1
k=i+1
= - + > i|+ (-m(1 - m)--i-1 [| (1
-eg.
(53)
roott pp + [1 [.
(54)
In this problem setting, assuming that 8M is sufficiently small compared to M, we can rewrite the error & as a linear function of 8M, which enables us to interpret the contribution of different pairwise similarities to the final error. The following theorem describes the exact decomposition of the impact of task order in this linear perturbation limit.
17

---

Optimal Task Order for Continual Learning of Multiple Tasks
Theorem B.2. Let us suppose that all elements of a upper-triangle matrix with zero-diagonal components 6M satisfies, 16Msyl 8m where 8m is a positive constant. Then, the error Eg is rewritten as below:
Ef Ey Pa) + Σ Σ
GyyMM + ((6m),
(55)
where
f, Pol =
M) MM Giz = go( go + go(i) + 9 +i) + g- i),
+11 [1 (1 pol1)
1/2 (I+ (1)-))
(56a)
(56b)
and functions 9o. --, Z - R that constitute the coefficient G are defined by
(k)) (1-pw)m
2-m (1 mmP-4 (po - 2-m
(1-polm
(1 - m) P+k-1
(57a)
9+ (s) (Po [1-po)m)
3-m (1 - -m)-1_ (1-
Pm + +2(1 - m) [--1² (1 - m) m
(57b)
2-m
2-m
2-m
2-m
gd(d)= = (1-Po)m 1 (1 -
(1-m)a= m)d-1
[po- - (1-po)m
(1 (1- (1-m) (
1--m + 3-m ))
mP
(1-po)m 1
(1 - m) P-d
2-m 2-m
2-m
2-m 1-m
(57c)
Matrix G specifies the relative contribution of each task-to-task similarity to the final error. Notably, Ey[m, pol term permutation invariant by construction. Thus, task-order dependence stems from the 8M -dddenendent terms. Inserting P. = 1 into Eqs. 57, you get Theorem 4.1 in the main text.
B.2. Proof of Theorem B.2
The inverse of (I M + 6M) is rewritten
( M + -4+ ( +
(58)
Thus, up to the leading order with respect to 8M, we have
Ex (po11" +11 (111poo) pp) 1/2 (T
M + +5M -1 (M (M (M+5M)
(00111 +[1-po]4) 1/2 (I + M) - 11 1M7
+ 2tr [M + M (Cou)) 1/2 (I + M) (6M - (1 M)-1 + 0(0M²).
(59)
The first term corresponds to ēy MM Cruu], thus it is enough show that the coefficients of 8MM is written as Eqs. 56b and
57.
M(I + - term is rewritten as
P [M + M) 11 = Σ
m lk > i] (5yx- m/k > -
(1- m)
k=1
=mlj 1]+ ((1 - (01- (1 m)) P))
= [j > i m (1 - [1 - (1 m) P P11 - mlj < +|]
- ((1-))-+ m)i-s- -- -(1--1111 -1-m) P-)
= m(1 - P mlj < + (1 - -
(60)
In the second line, we defined k5s by ky = max(i + 1, + 1). For the ease of notation, let us denote the last term in the equation above as
Uy m(1 m) mli s (1+ m)²
(61)
18

---

Optimal Task Order for Continual Learning of Multiple Tasks
Next, (0o11² +[T-pol1 (I M)-1 term becomes
[po111 - - (1 + (1)
=E6o (Po+ + po)ost) (ōus mli k)+(11 m))-k-1)
1 ( Σ mlj > k3| (1 m) m)i-k-1 + (1 - [p) (8.g - -mlj +)+(1 m)--i-1)
Po
= Po(1 - -m)-- + (1 - Po)oij - (1 - --ommmj > 1)+(1 m)
(62)
In the last line, used Σx mlj x1+(1 mm5-k-11 (1 m)-- As before, let denote the coefficient by
Wij Po - 10-1-m3-1 - po)mli (1 m)-i-1
(63)
Then, the first-order term with respect 5M follows
tr [A + +M- (po11 + (1 (o))) (1 -M-- (6M² 5M M))-
= =E (M [10011 - ([o-) (T + 1, (Mii - 2oMb [M + M7)
isk
Σ Uzi ((1 - po)00a + usx)) (MMin Σ MMari
Σ 8MM Σes ozf (dix 1(1 po)dji + ugi] vu 1(1 po)ōgN jk + uzzl))
= Σ
GRI
(64)
where the coefficient (k: 1)-th element is defined by
Gı Eese Use (8ek [[1 po)65r ual wis [(1 po)6GE uzzs)
(65)
GkI is decomposed into
GkI = - -Po)uu + Σ Eouuu - (1 (1-pal po) Σ Uht Σ Phonuak
(66)
The first term (1 Pi) is rewritten as
(1 Po)utk = (1 - po)m(( (1 - Pm mll < k]+ (1 m)--1
(1 - po)m(1 m) P-1 m)
(67)
Here, we dropped the second term, because 8M when S
19

---

Optimal Task Order for Continual Learning of Multiple Tasks
Regarding the second term, summation over j evaluated
P
Σ (m(1 -m)P-s - -mlj - 4]+ m) (po(1 mm)-1 - (1 - Po)ma(k > 5]+ -m)*--1)
PjiMijk
=po(1 mm)-1
m Σ
11 m)
P-j m)mmm
m Σ m)
j=1
jik
-(1 Po)m²?
Σ (1-m))(P-)++e ())()))+(k---))
Σ (1 -m) (0)--)+(6---1)
1=1
j=1
Po m) k-
(11 m)² (1 - m) (1-
(1 Pa)m
([(1 - m) P- (k- (1 m) m) (k--)) - [[1 -m))tt---22) m))th-11)
2 - m
(1- - (1-pa)m
((1 m) m)) - (1 - m) P+k-1) + (1-pa)m
2 (1-
- (1- (1-m))--))) m))-(A-1))
Po 2 m
(68)
In the third line, we defined jik as jux = min(k: 1,)
Thus, the second, E, Uyb term becomes
U5kUgl = (1p-- (1- (1-po)m
(1 - (1m 1-m))+(-11 + (1- - pa)m
Σ )
P+1-1
(1 m) mmP-11 P-1+1
2 - m
2 - m
(1 pa)m
(1 - m) - + (1 - Po)m
Po
(1 m))---1
(69)
2 m
m
The third term is, from similar calculation, rewritten as
Ut Uki
Σ (m(1 - -m) --m ml| < (1 - m))-t) (mm1 - -)) - mm(k < +)
-m)r-k)
P
-Pm²(1-m1) -m))pP(444) - m² (1 (1-m)) m) P-R Σ (1 -m)² m))-i + (1
p-
P-1 Σ (1 m)imh
+ m² Σ
(1 m) mi (k++2)
im1
imk
iml
= ((pm² + 22m(1 m) m(1 m)²
(1 m) 2P (k+1) m ((1 m) P- m)P-h +
(1-m)
m) p) +
(1 11mm)=
m
m
2 m
(70)
The last term is a little more complicated, so let us divide it into two terms:
(1 Pa)m
(1 1-Po)
Σ U/s Σ
Po
Ti
T2
(71)
2 m
2 m
where
P
P
T = m(1 -m) P-1
((1 - m) + 1(1 (1- P+k-1) m >
(1 --mmm)- (111-m)tt-1-- (1- (1-m) P+k-1),
1=1
i=1
p
T2 m1 m)) P-1
(1- (1 m) (k--11
Σ (1 m)=- (a1 (1-m))-(kk1) m) P-(--(()))))
i=!
(72a)
20

---

Optimal Task Order for Continual Learning of Multiple Tasks
Taking summation over index i, T1 and T2 are rewritten as
Ti = mm - m) P-1 (1 (1-m) m)*
[] - (1-mmm m) - PP(1 - m) m))th-l))
m
m1))((th11 - (1 - m) -m)2p+++((-))) - (1-m)p-te
[1 (1 - m) m)pt1-1))
2 - m
1 - (1 m)
2 1 - m 2 m ) (1- - m) m) (1
mP
1 m
1
(1 - m) + + (1 - p
(73)
2 - m
T2 = (1 - m)
P-1 (11 (1 m) - + [[1 m) (1 - m) P mP - (
1
[ m))-k+3 (1 m) 22)2333-+++ [| (1 m) 1+1 m) -)--(k-1) (k-1)
2 m
1 1- m (1 m))
1 (1 m) )) P-(1-N) ((2 (12-m)(( m1 m))ppppp---- + (1 m) m)P-h+1)
1
P-1
2 m
(1 m)
(1 - m) 2P (k+1)
(mP(1 - m)) + 2(1 m)²
(74)
2 - m
The results above show that Ga is decomposed into four components:
Ga = g- (1- k)) + g g- + k)) + gg (k) + gn(1).
(75)
Summing over the terms that only depends on k; we have
gr (k) = (1 Po) (-m(1 - m) - (po- (1-polm (1 c1-m) mm-m)=tt Ptk- (1-pa)m
2-m (1 m) -
(1 (1 - p - (po (1-po)m) (1 m) P4K-1
(76)
2-m
Similarly, the terms that only depend are summed up to:
gn((1= (1 1 -Po)m( -))--i- - 0o (1-pa)m
(1 -mm)+tt P+1-1 + 2-m
(1-po)m (1 m))-1+1)
2-m m
- -(1- - Po) (-m(1 - m) p-1- 1l--a)mm
(2 - m1) - -m) P-1
2-m
(1-po)m (1 m) - (po
2-m
(1-pa pm (1 m) P
2-m
(77)
Therefore, 9r. and In have the same form. We denote this function g. below.
9- (7 + k) term has a slightly more complicated expression:
9+(1 + k) (po- (1- 2-m
(1 m)++l-1 (1 - po)m (Pm + 2(1 m) (1-m)²
2-m (1 - m)2P-(k+11)
+ (po (1-po)m)
(1-pa) pm- (1 (1)) (Pm + +2(1- m)
[1-m]²
2-m (1 m))P-(k+!l)
2-m
2-m
2-m
Po (1- pm
3
3-m
m) t) (1-p))m 2-m
(Pm + 2(1 m) [1 [1-m)²]
-
(1 m) (k++)
(k+1)
(78)
2
Lastly, g- (1 - k)) term becomes
g- (7- 1 k))
(1-pe)m
(1 m)² k- (1 (1-polm 2-m
(1 - (1-m)²- -(po -pp (1-po)m) ) (1 (1 m)² (2+
mP 1-m
1-m )) (1 - P-(MM)
2-m
2-m
(1-Pol) pm)
1-m (1 m)/-* + I-m (1 - -m)+ (-(()))))
2-m
2-m!
(1 )m
(1 m) m - [(1 (1-(1) (1 m)"
I-m + 2-m)
3mm) ))
(1-paJm 1 (1 m 2-m
(79)
2-m
2-m
1-m
We thus obtain Eq. 57. If we set Po = 1, we recover Theorem 4.1. In Theorem 4.1, G is defined as G = g+ + +) + (i) + 9n()).
21

---

Optimal Task Order for Continual Learning of Multiple Tasks
Periphery-to Core-lo periphery
Max path Min path
10
Core-lo-perpherry Average Periphery-lo-oon
b 12
Min path Average
1.0
1.0-
0.9
Error 8.
path
0.8
Error
Ratio
0.5
Ratio
0.7
0.6
P
P= 7
0.5
0.2
P 5
P 5
P 3
P 3
-0.1
0.1 0.2
0.2
Average correlation m
Average correlation m
Average correlation m
Average correlation m
Figure 6. a, by The same as Fig. 3e and g. but without normalization. The average correlation dominates the enror but the order dependence is observed robustly. e) The ratio of networks where the periphery-to-con order outperforms core-to- core-to-periphent periphery order. As before, we generated 1000 samples of C matrices randomly while fixing Crome to be all one. then binned C based on the average off-diagonal correlation. For each average correlation value m, then calculated the ratio of networks in which the periphery-to-coe order achieved smaller error than the core-to-periphery order. d) The same as panel c, but the ratio of random seeds where the max- max-path orders achieved the better performance than the min-path orders.
B.3. The Impact of Task Typicality
We can further analyze the impact of task typicality on the task order discuss in the main text. Let us define 9 by
P
m)a = 1
g = P
Σ
mP ((1 m) (1 -m)++))
(80)
1=1
then G is rewritten as
(3-m)
(1 m) P-1] [[1 m)) + (1 - m)
uv
22 12-m][1
3-m
((1 m)" -
- g) ((1 - m)) m g) (-m)] m mm
(81)
2-m)(11m
m)
Thus, the corresponding error term oe becomes
5f = Σ Σ
G
p=1 v=μ+1
p
(3- (3-m)g (2-m)(1 IIm
(1 (1- P-1
Σ (1 mm)"8l,.
p
P Σ ((1-)) H g) ((1 (1m 3) 6M + const.
+
3-m
(82)
m
#=1 v=
If (3-m)ğ (1 P-1 from the rearrangement inequality, the first term is minimized under öt, I 8tz ... ... 8tp
(2-m)(1-m)
ordering. However, the second term may not be minimized under this task order.
C. Implementation Details
Source codes for all the numerical experiments are available at https://gtttubb.oom/iaaas wttp://......ioioae..e/. -i- 1i- aotimal-learn-on
code/ coceee/e
C.1. Linear Teacher-student Model with Latent Variables
In the simulations depicted in Figs. 2b, 3eg, and 4ef, we set the latent vector size N, = 30, input layer size N₂ = 3000, and the output layer size N, = 10. We initialized the weight matrix W as the zero matrix, and then updated the weight using gradient descent (Eq. 21) with learning rate 77 = 0.001 for 100 iterations per task. In Fig. 2b. the input correlation matrix Cn PxP was generated randomly in the following manner. First, we generated a strictly upper-triangular matrix
22

---

Optimal Task Order for Continual Learning of Multiple Tasks
CIFAR-100 (5-to-1 classification)
CIFAR-10 (training-based similarity)
0.42
6.40
C 0.3
Pertphery-to-core
Average Core-lo-periieeee perr-leyperippeeyy
Performance
Perforance
6:36
Performance
Perforance
0.7
0.50
0.30
0.6
Periphary-to-com tororr Average Core-to-priiherry
Max path Average Min path
Max path
0.5
0.3
Average
Min path
0.20
-0.04
-0.03
5.00
0.02
5.04
0.50
6.20
-2.04
-0.02
0.00
0.22
0.04
0.06
0.2 Average similarity
0.3
0.4
-b3
c.2 Average similarity
0.0
0,1
0.3
Average similarity
Average similarity
Figure 7. Task order preference in continuous image classification tasks. a.b) Average classification performance after continual learning of CIFAR-100 where each task consists of 5 label classifications. Here, we randomly picked 25 labels from CIFAR-100 dataset and generated 5 tasks each requiring classification of labels. c,d) Average classification performance after continual learning on CIFAR- 10. Task similarity estimated by evaluating zero-shot generalization performance from task A to B, using the training data of B instead of the test data (see Appendix C.4 for details).
C by sampling each element independently from continuous uniform distribution between 0 and 1, Mo.11- We then generated the full matrix C"" by C C'i,U'" + (Cii..Ui" + I. If the resultant C is positive semi-definite matrix, we accepted the matrix, otherwise, we generated C in the same manner, until we obtain positive semi-definite matrix. Here, we limited the correlation to be positive mainly because continual learning is typically impractical when there exists a large negative correlation between tasks. We generated Coue using the same method. In Fig. 2c, we estimated the final error εy using Eq. for each triplet (PAB, PBC, PCA). We calculated the error for all six task orders and plotted the order that yielded the minimum error.
In Figs. 3e and g, we generated input correlation matrix C using the same method with Fig. 2b, but we instead sampled the elements from uniform distribution between -1 and 1,) The average correlation m was defined by the average of the off-diagonal components of Cr The output correlation matrix Coutt was set to be the all one matrix (i.e. Cove = 1 for all (n, v) pairs) which corresponds to the P. = 1 scenario. We estimated the error under each task order for 1000 randomly generated input correlation matrices and binned the performance by the average correlation. The average performance (black lines in Figs. 3e and g) were estimated by taking the average over randomly sampled 100 task orders. The error bars, representing the standard error of mean, are larger for larger average correlation because we didn't generated many C with large average correlation under our random generation method. Because there are two task sequence that provides the max- max-path due to symmetry (e.g. A A-C-E-B D and D-B E CCAA in Fig. 3c), we defined the error of the max-pati rule as the average over these two task orders.
C.2. Generation of Input Similarity Matrices Having Simple Graph Structures
In Fig. 4, we introduced simple graph structures to the task similarity matrices. To this end, we first generated unweighted bidirectional adjacency matrix Aaj E (0, 1) PxP given a graph structure, then calculated distance between nodes on the graph specified by Ags, which we denote as D. From this distance matrix D, we generated the input similarity matrix Cr by C = a Pa for each task pairs (i. j). Here, a is the constant that controls overall task similarity. a R 1 means that inputs for all tasks are highly correlated, whereas a x= 0 means that they are mostly independent.
For instance, in the case of chain graph, the adjacency matrix is given by Adj.aj = 1 if j = i + 1 else Agjasj = 0. Thus, the distance between nodes D follows D.j i - jl and the input correlation matrix becomes Ci₁y ali- In the ring graph, the distance between node is instead given by D,j min jl. P j j|}. For the tree graph, we used tree where each lonfleaf node has exactly two children nodes. The same structure was assumed for the leaves graph, except that we only used leaf nodes for constructing the task similarity matrix.
C.3. Convolutional and Multi-layered Non-linear Neural Networks for Image Classification
We used convolutional neural networks (CNNs) for numerical experiments with the CIFAR- 10 and CIFAR-100 datasets. The network consisted of two convolutional layers and one dense layer, followed by an output layer. The first convolutional layer had 32 filters with 3 x 3 kernels. The output was passed through a Rectified Linear Unit (ReLU) activation function and then downsampled using average pooling with a window size of 2 x 2 and a stride of 2. The second convolutional layer was similar to the first, except that we used 64 filters. The dense layer following the two convolutional layers had 256
23

---

Optimal Task Order for Continual Learning of Multiple Tasks
Zero-shot transfer performance
Task similarity matrix
1.0
0013
0.59
4.09
1.03
2.89
0.38
0.18
3.13
0.55
2.02
0.31
4.38
4029
40.31
0.83
.0.5
-0.12
-00
0.84
1:56
6.77
9.15
2.90
0.37
-0.3
0.83
0.04
-0.5
1.06
1.78
1.99
1.29
0.30
-1,0
Figure Schematic of the task similarity estimation. From the error in zero-shot transfer 6u [Wi] (left), we estimated task similarity Par (right).
a 0.8
0.1% data
C
1% data
10% data
100% data
Periphery to core Average
08
Periphery care
0.4
Periphery core Average
0.8
Periphery to care Average
Performance
Core-fo- perriphhrrrr
Performance
C.r- sereett-ppene periphey Core
Average
Performance
Care Care-to-peripaeee to-periphery
Performance
0.7
Core-to-pae periphery
0.6
0.6
0.6
0.5
94
-6.1
2.0
0.1
0.4
-0.1
5.0
0.1
6.
0.4 -5.1
0.0
6.1
0.2
63
0.4 -0.1
5.0
0.1
0.2
03
Average similarity
Average similarity
Average similarity
Average similarity
b
f
h
o.8
path Average
path
0.8
path
path
Average
Average
Average
Performance
bur
Min
Performance
0.7
Min palh
Performance
9.7
path
Performance
path
b.s
0.6
9.6
2.6
0.s
0.5
0402 0.1
0.0
0.1
0.4
0.1
0.0 0.1
0.2
0.4
0.0
0.1
0.2
0.6
-0.1
0.1 0.2 0.3
Average similarity
Average similarity
Average similarity
Average similarity
Figure Task order preference estimated from various fraction training data under CIFAR- 10 task. For instance, 0.1% data results depicted in panels and b, we first used 0.1% of training data (~ 10 images) for training the network with one task and used 0.1% 1% of training data from another task to evaluate the zero-shot transfer performance. We then estimated the task similarity as before. Panels g and h the same with panels and d in Fig. 7. We used the same set of parameters with the results depicted in the main figure (sec Appendix for details).
neurons, with ReLU as the activation function. For classification, we used a softmax activation function in the last layer. The weights of both convolutional and dense layers were initialized with LeCun normal initializers.
For the -MISST dataset, we used multi-layer perceptrons (MLPs) evaluate the robustness of our findings against the neural architecture. The MLP model had two hidden layers with 128 and 64 neurons, respectively. We used ReL.U the activation function for the hidden layers and softmax for the output layer.
In both CNN and MLP models, we studied binary classification with two output neurons, except in Figure 7a and b, where we considered a classification of labels with five output neurons. All tasks were implemented as single-head continual learning where the output nodes are shared across tasks. The performance was evaluated by the average classification accuracy on the test datasets for all the tasks at the end of the entire training.
The networks were trained by minimizing the cross-entropy loss using the Adam optimizer with learning rate of 10-3 for five epochs per task. We set the batch size to 4 due GPU memory constraints. The models were implemented using Flax (Heek et al., 2024), JAX neural network library, and were trained on NVIDIA Tesla V100 GPUs
In both Figures 5 and 7, we generated 100 task sets by randomly dividing 10 labels into a set of five binary classification tasks. In the case of CIFAR-100, we initially sampled 10 labels randomly, then partitioned them into five binary classification. We binned the task sets by the average similarity estimated from Eq. 16, then plotted the mean performance and the standard error of mean for each bin. The black lines representing the average performance of random task order were estimated by
24

---

Optimal Task Order for Continual Learning of Multiple Tasks
a
Fashion MNIST
CIFAR-10
CIFAR-100
Periphery-to-core Core-o-perrihery
Periphery-lo care
Periphery-lo-corg Average Core to periphery
Periphery-to-coe Average
Performance
0.8
Average
Core- to periphody
Perforance
0.7
Performance
0.7
Core-to-prribhery
0.8
3.7
Ratio
0.6-
0.6
0.6-
2.5
0.5
0.5
0.2
p=
5
P=
3
04
-0.1
0.1
.0.3
Average similarity
Average similarity
Average similarity
Average similarity
f
h
Max path Min path
0.8
Max path
Max path
Max path
Performance
c.
Average
Min path
Performance
Average Min path
Performance
0.7
Min path
Average
6.7-
Ratio
0.6
P
02
P a 5
P 3
0.6
-0.1 0.0
0.3 0:4
0.6
-0.1
0.0
-0.1
0.0
0.0
-0.2
0.0
Average similarity
Average similarity
Average similarity
Average similarity
Figure 10. Task order preference continuous image classification tasks where the task similarity was estimated from 1% of training data, as opposed to Fig. 5 where 100% of data was used. a-f) Continual learning performance, defined as the average test accuracy across all the tasks after learning, under various task orders. g, h) The ratio of task sets where the periphery-to-oe -o-roor rule outperforms the core-to-periphery rule (g), and where the max-path rule outperforms the min-path rule (h), under CIFAR-100 with different numbers of tasks (P = 3, 5, 7). Panels and were replicated from Fig. for completeness. Note that task-order effects were smaller in CIFAR- because the number of training images per label is fewer in CIFAR- 100 than in CIFAR- 10 and Fashion-MNIST.
taking the mean of the classification performance under 30 random task orders for each task set. Because there are two task orders that provides the max max-path by construction, we define the performance of the max-path rule the mean of performance under these two task orders.
C.4. Estimation of Task Similarity
In the main text, inferred similarity between two tasks A and using zero-shot transfer performance between the two tasks. Previous work linear model indicates that, if the output similarity is one, the pairwise transfer performance ACTply [=] = En [W,] Ej [W] is written as (Hiratani)
AeTF(V μ4 P/2 pm))
(83)
indicating that the input similarity between two tasks can be inferred as
P
-errer -> n4).
Aetrlu
(84)
Motivated by this relationship, we defined similarity between tasks in general nonlinear networks by Eq. 16. A similar approach was implemented in (Lad et al., 2009). Notably, this method only requires inputs/outputs of the trained network, and thus applicable situation where the model details are inaccessible (e.g., human and animal brains, closed-LLM).
We implemented the evaluation the zero-shot transfer performance from task A to B as follows: on task A for 5 epochs from a random initialization using the Adam optimizer on the cross- cross-entropy loss with learning rate 10-3 as above. We then measured the cross-entropy loss on the test dataset of task B, f WA), where WA represents the weight after 5 epochs of training on task A. To normalize the accuracy, we divided the obtained loss by the loss on task B under a label shuffliigg <B A] The resultant value <nnWW.] <B..r characterizes how well the network transfer to task B compared random task with the same input statistics. Since evaluating similarity using the transfer performance on the test data may potentially introduce bias, in Fig. 7c and d, we estimated the transfer performance Ae [Wa] using the training dataset for task B. Even in this setting, we found results nearly identical to those in Fig. 5c and d, confirming the robustness of our findings with respect to the details of the similarity evaluation method.
First, we trained network
In the task similarity and order estimation from sparse data shown in Figs. 9 and 10, we estimated task similarity in the same manner as described above, but using subsampled training data. For example, the 1% data scenario, we used 1% of
25

---

Optimal Task Order for Continual Learning of Multiple Tasks
the training data for task A calculate the weights after learning task A, denoted WA- As before, trained the network using the Adam optimizer on the cross-entropy loss for 5 epochs from random initialization except that we only used 1% of training data at each epoch. We then estimated the transfer performance to task B, e [Wa]. using 1% of the training data for task B. The task similarity between tasks A and B was then computed using Eq. 16, as in the full data setting.
26


Reviews:
[{"review_id": "OLeKZ772TN", "reviewer": "Reviewer_Z4Kz", "summary": "The paper considers task orders in continual learning. It formulates a linear teacher-student model and derives an explicit expression for the errors in the asymptotic case. Then the paper analyzes how the change of task order would affect the error.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments on our work. Please find the replies below (we quoted your comments only partially due to space constraint). \n\n**Claims And Evidence**\n\n> For example, the title is...\n\nPlease note that, both in the abstract and main text, we noted that these principles are based on linear models and apply only to specific parameter ranges.\n\n> The paper also claims that it proposes two task ordering rules,...\n\nWhile Bell \\& Lawrence (2022) is an important inspiration, our work represents a significant advancement beyond theirs.\nFirst, we develop a theoretical framework to explain how task similarity and task order impact continual learning performance, in contrast to their primarily empirical approach. Specifically, our analysis sheds light on why the max-path on a task dissimilarity graph often improves performance and how the number of tasks influences this trend.\nIn addition, we introduce and analyze an alternative heuristic for task-order optimization: the periphery-to-core rule.\nFinally, unlike Bell \\& Lawrence (2022), our task similarity inference method does not require access to model parameters.\n\n**Other Strengths And Weaknesses**\n\n> The paper advocates using a linear teacher-student model. However,...\n\nThe variables $\\boldsymbol{x}$, $\\boldsymbol{y}$, and $\\boldsymbol{y^*}$ represent the input, output, and target, and are essential for defining the problem. The latent factor $\\boldsymbol{s}$ allows flexible control over task similarity in the input and output spaces ($C^{in}$ and $C^{out}$). This setup enables us to examine how input-space similarity, $C^{in}$, influences continual learning performance and optimal task order, which previous linear models (e.g., Lin et al., 2023) do not capture.\n\n> The distribution assumption such as (2) is not very well motivated.\n\nThis specific assumption was introduced for analytical tractability. However, we later showed that task similarity can be estimated from data, and its effects on task ordering and continual learning performance are consistent with our theory.\n\n> The paper uses the symbol of inner product to denote expectation (see Eq 3). \n\n$\\langle x \\rangle$ symbol is commonly used to denote the expectation of $x$ in the field of mathematical sciences and machine learning works employing mathematical physics methodology.\n\n> Then, also in Eq 3, $\\epsilon$ is defined as ...\n\nWe defined the error on task $\\mu$, $\\epsilon_{\\mu}$, as a function of the model weights. We then derived the average error, $\\bar{\\epsilon}_f$, in terms of the task correlation matrices to analyze how task correlations impact performance. In Theorem 4.1, we focused on uniform correlation matrices parameterized by $m$ and $\\rho_o$, up to a small perturbation term, and denoted the corresponding error as $\\bar{\\epsilon}_f[m, \\rho_o]$.\n\n> The symbol $\\delta$ was used at Line 160,...\n\nThank you for pointing out. The definition of delta was indeed missing at L160. Here, $\\delta_{ik}$ is defined as the Kronecker delta, which returns 1 if $i = k$, and returns 0 otherwise. \n$\\delta M$ is a matrix, as noted in Theorem 4.1, and $\\delta M_{\\mu\\nu}$ is the $(\\mu,\\nu)$-th element of matrix $\\delta M$. $\\delta_m$ is a small positive constant defined as the upper bound of $\\delta M_{\\mu\\nu}$. We will clarify these points in the revision. \n\n> While it is understandable that finding an optimal task order is difficult,...\n\nA small perturbation does not imply a small final error; rather, it implies a small change in the final error. This allows us to use a first-order approximation to analyze how slight changes in task similarity affect the error.\nFurthermore, we did not optimize $G_{\\mu\\nu}$. This coefficient quantifies how a small change in the similarity between the $\\mu$-th and $\\nu$-th tasks influences the final error.\n\n> At Line 220 (right),\n\n$\\delta M_{\\mu\\nu}$ captures the relative similarity between tasks $\\mu$ and $\\nu$, thus its summation $\\sum_{\\nu\\neq\\mu} \\delta M_{\\mu\\nu}$ represents how similar task $\\mu$ is with all other tasks, which in turn measures the typicality of task $\\mu$. \n\n> At line 267 (right),...\n\n$C_{\\mu\\nu}^{in}$ captures the correlation between tasks $\\mu$ and $\\nu$ in the input space (see Eq. 2), and thus $d_{\\mu\\nu} \\equiv 1 - C_{\\mu\\nu}^{in}$ represents the dissimilarity.\n\n**Other Comments Or Suggestions**\n\n> At Eq 5,\n\nIn our setting, there are infinitely many solutions that minimize task-specific error. We focus on the solution reached by gradient descent, as it is the standard optimization method for neural networks. While convergence technically requires infinite iterations, the convergence rate is fast due to convexity.\n\n> The description \"periphery-to-core\" is very confusing,...\n\nWhile Figs. 3b and c are illustrations, they portray the multi-dimensional scaling (MDS) visualization of the task similarity matrix. For instance, in the chain-like task similarity depicted in Fig. 4a, task C is central whereas tasks A and E are peripheral.", "claims_and_evidence": "Some claims made in the submission are not supported by clear and convincing evidence.\n\nFor example, the title is \"Optimal Task Order for Continual Learning of Multiple Tasks\". This is an overclaim, as the paper has not proved any optimality regarding task orders. The lack of serious proofs invalidates the claim that \"this paper proposes two principles for task order optimization\". The use of various heuristics under conveniently simplified settings is a contradiction to the word, \"principle\".\n\nThe paper also claims that it proposes two task ordering rules, the periphery-to-core rule and the max-path rule. This is also an overclaim, which the authors themselves actually know and cited: The max-path rule was considered in prior work of Bell & Lawrence (2022). Indeed, many experimental findings and figures follow from Bell & Lawrence (2022) in spirit.", "methods_and_evaluation_criteria": "The proposed formulation under the linear teacher-student model does not make too much sense (see comments below).", "theoretical_claims": "I didn't check the correctness of the proofs.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "I am not sure.", "essential_references_not_discussed": "I am not sure.", "other_strengths_and_weaknesses": "## Linear teacher-student models\n- The paper advocates using a linear teacher-student model. However, this viewpoint is questionable, as it only introduces notational complexity and never provides new insights. Variables $s, x, y, y^*$ later never appear in the paper. At the end of the day, the objective the paper aims to minimize is (3), which is a least-squares objective. From this viewpoint, the setting has little difference from prior work focusing on linear models (e.g. Lin et al 2023). By directly introducing objective (3), the paper would be much more readable.\n- The distribution assumption such as (2) is not very well motivated. Why would the data of different tasks be related in such a specific way? Practical examples are needed to justify this assumption.\n\n\n\n## Clarity and Motivation\nThe clarity of the paper is largely compromised due to the use of non-standard notations. For example:\n- The paper uses the symbol of inner product to denote expectation (see Eq 3). Then, also in Eq 3, $\\epsilon$ is defined as an error function of weight $W$, and $\\overline{\\epsilon}$ is defined as the minimum average error, but this is now a function of correlation matrices. But later in Theorem 4.1, $\\overline{\\epsilon}[m,\\rho\\_o]$ seems to suggest that $\\overline{\\epsilon}$ is a function of two numbers, $m$ and $\\rho\\_o$. \n- The symbol $\\delta$ was used at Line 160, and its meaning is not defined. This is confused with $\\delta M$, which means the perturbation. The confusion deepens when we see $G_{\\mu \\nu} \\delta M_{\\mu \\nu}$ in Theorem 4.1, while Theorem 4.1 contains another $\\delta$, which is $\\delta_m$, some positive constant. \n\nAt various places, the definitions of certain quantities lack motivations as well. For example:\n- While it is understandable that finding an optimal task order is difficult, the introduction of linear perturbation theory needs a better motivation. Why could a small perturbation indicate a small final error? If this issue is not addressed, going ahead and optimizing $G_{\\mu \\nu}$ appears to be very heuristic. \n- At Line 220 (right), why does $\\delta t_{\\mu}$ defined in such a way indicate \"relative typically of the task\"? \n- At line 267 (right), why does $d_{\\mu \\nu}$ defined in such as way indicate \"dissimilarity\"?", "other_comments_or_suggestions": "- The paragraph below Eq. 4 seems to have some formatting issues.\n- At Eq 5, since the current task is solved optimally, why should one consider gradient descent specifically? gradient descent couldn't find the exact solution (5) unless it is run in infinitely many iterations\n- Line 195 (left): permutation-invatiant\n- Line 188 (right): Grammar at sentence \"If $\\rho\\_{AB}=1$ ...\"\n- Line 271 (left): \"mroe\"\n- Section 4.3 should point out that finding the longest Hamiltonian path in a (weighted) graph is NP-hard.\n- The description \"periphery-to-core\" is very confusing, and I found Figure 3c not useful. The tasks defined as per the paper have no \"spatial\" relations. In the caption, why tasks A-C are central and D-I are periphery? Is that because they are plotted that way?\n- Line 423 (left): \"bu\"", "questions_for_authors": "See the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhaPnqLcpH", "reviewer": "Reviewer_r3vn", "summary": "This paper investigates the impact of task order on learning performance in continual learning. It first mathematically models the correlations between tasks and then derives two rules to enhance learning effectiveness: (1) learning non-representative tasks before typical ones, referred to as the *periphery-to-core rule*; and (2) ensuring that consecutive tasks are as dissimilar as possible, known as the *max-path rule*. Experimental results validate the effectiveness of these principles.\n\n### Update after rebuttal\nIn summary, this paper has certain limitations in real-world applications. Maybe it is useful in some cases (e.g., the self-driving one you mentioned), but it is less practical than the existing setting of continual learning. However, I think this paper is a good and insightful attempt, so I will keep my initial recommendation, leaning towards accepting this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. Please find our replies below. \n\n> it lacks a clear table summarizing the performance impact of applying or not applying the two proposed rules.\n\nThe table below summarizes the results from Figs. 5a-f. \n\n|   | Fashion MNIST| CIFAR-10     | CIFAR-100    |\n|:--:|:--:|:--:|:--:|\n| Peri-core (\\%)  | 67.25 ± 0.98 | 64.35 ± 0.54 | 63.96 ± 0.52 |\n| Max-path (\\%)     | 66.03 ± 0.88 | 62.20 ± 0.56 | 62.61 ± 0.48 | \n| Min-path (\\%)     | 56.96 ± 0.81 | 55.55 ± 0.51 | 57.50 ± 0.40 |\n| Core-peri (\\%)  | 51.32 ± 1.16 | 51.15 ± 0.68 | 55.02 ± 0.57 |\n| Control (\\%)    | 61.97 ± 0.78 | 59.10 ± 0.47 | 60.16 ± 0.38 |\n\nHere, we took the average over 100 random seeds, with errors representing the standard error of the mean (we crunched the x-axes of Fig. 5). This table confirms the robust advantage of the proposed rules.\nIn Fig. 5, we depicted it in graph, as we identified mean task similarity, used as the x-axis, as a significant covariate.\nWe will include the table in the revised manuscript.\n\n> In scenarios such as LLM training and robotics, while data may be processed in steps, each step typically follows a similar distribution.\n\nWe believe our framework remains applicable even when tasks share similar distributions, as you may see in LLM training or robotics. \n\nTo illustrate this, we applied our algorithms to Fashion-MNIST, using a random split of the training dataset into five sub-tasks. Specifically, we selected 500 images from the training set and created five sub-tasks, each containing 100 randomly chosen images. We considered a 10-class classification task here, as each sub-task included all 10 labels.\nWe estimated the similarity by measuring the zero-shot transfer performance between sub-tasks using Eq. 16. Since the data was randomly split, all sub-tasks were inherently similar, but it varied across them.\nWe then trained a network sequentially on the five sub-tasks using the inferred task orders. We found that task orders determined by the periphery-to-core rule and the max-path rule yielded better test accuracy than the control (see the table below). We observed the same trend with CIFAR-10, although overall performance was low in this low-sample regime.\n\n|   | Fashion MNIST |  CIFAR-10  | \n|:--:|:--:|:--:|\n| Peri-core (\\%)  | 77.25 ± 0.19 | 31.70 ± 0.21 | \n| Max path (\\%)   | 77.03 ± 0.13 | 31.67 ± 0.18 | \n| Min path (\\%)   | 76.35 ± 0.15 | 30.64 ± 0.18 | \n| Core-peri (\\%)  | 76.26 ± 0.25 | 30.24 ± 0.27 | \n| Control (\\%)    | 76.63 ± 0.10 | 31.01 ± 0.13 | \n\nIt remains unclear whether it scales to LLMs, where both the corpus size and input space are vast (due to the sequential nature of the input). Nevertheless, it highlights the potential utility of task-order optimization even when tasks follow highly similar distributions.\n\n> I would like the authors to clarify whether there exist application scenarios where (1) all tasks are fully known beforehand, yet (2) it is still necessary to learn only one task or a subset of tasks at a time.\n\nRegarding (1), please note that our framework does not require access to the full dataset for all tasks in advance.\nTo clarify this point, we newly conducted an experiment where we used only a small subset of training data for estimating the task similarity and the optimal order (here we considered five binary classification tasks as in Fig. 5). \nThe table attached to the reply to Reviewer Z3Hz describes the average performance of various task orders on split CIFAR-10 in this task setting. We found that even when using only 1\\% of the data, the max-path and periphery-to-core rules consistently outperformed the random task order (please see the explanation below the table for details).\n\nThis sufficiency of partial observation opens up many applications, because it enables a researcher to first collect a small amount of data for all tasks, then perform online learning using the inferred task order while gradually acquiring the full dataset.\nFor example, consider training a self-driving algorithm across three distinct environments. \nDue to geographical and economic constraints, it may be impractical to conduct simultaneous multi-task training across all three locations.\nIn such cases, a practical strategy is to run small-scale pilot experiments in each environment, measure task similarity, and determine the most effective order for full-scale training. Online learning can then proceed sequentially in each environment, without the need to wait for complete data collection from all locations.\nA similar strategy could be effective in medical image analysis, where data might need to be collected across different regions and diseases one by one.\n\nRegarding (2), in robotics, switching between tasks often involves physically rearranging objects around the robot, which is both time-consuming and labor-intensive. Thus, switching tasks on a trial-by-trial basis may be impractical, necessitating training on each task one at a time in a block-wise manner.", "claims_and_evidence": "Based on my understanding, the claims in this paper are well-founded and supported by rigorous reasoning.", "methods_and_evaluation_criteria": "From a research perspective, the proposed method is innovative and carries some significance. However, I have concerns about its practical applicability. In real-world continual learning scenarios, task order is typically not controllable on demand. Continual learning simulates the process of continuously updating a model in response to updating environment. Since these changes are unpredictable, the learning order cannot be predetermined. As a result, the practical relevance of this work may be questionable.", "theoretical_claims": "Based on my understanding, I did not find any theoretical claims that appear to be obviously problematic.", "experimental_designs_or_analyses": "The paper presents sufficient experimental results with relatively thorough analysis. However, it lacks a clear table summarizing the performance impact of applying or not applying the two proposed rules.", "relation_to_broader_scientific_literature": "The contributions of this paper are primarily within the field of continual learning. It may also be applicable to other continual learning scenarios beyond image classification.", "essential_references_not_discussed": "I did not observe such cases.", "other_strengths_and_weaknesses": "The strengths of this paper lie in its rigorous derivations and its novel perspective on continual learning tasks through task ordering. However, as noted in the **Methods and Evaluation Criteria** section, I have some doubts regarding its practical utility. Additionally, the paper lacks a clear table presenting intuitive results to demonstrate the effectiveness of the proposed method.", "other_comments_or_suggestions": "In the third paragraph of Sec. 1, the authors provide several examples to illustrate scenarios where, even when all tasks are known in advance, it is necessary to learn only a subset at a time to reduce learning difficulty/computational cost. This argument appears to counter my concerns about the method’s limited practicality. However, I find the authors’ reasoning somewhat unconvincing.\n\n\nIn scenarios such as LLM training and robotics, while data may be processed in steps, each step typically follows a similar distribution. This differs from the continual learning setting, where each step introduces a distinctly different task. Using the authors’ example of Task A, Task B, and Task C, these scenarios are more akin to training on one-third of A’s data, one-third of B’s data, and one-third of C’s data simultaneously at each step, rather than sequentially learning A, then B, then C. As a result, these scenarios do not inherently involve issues like catastrophic forgetting, which continual learning methods aim to address, making the proposed approach less relevant.\n\n\nThe real risk of catastrophic forgetting arises in scenarios where future tasks are unknown in advance. However, the proposed method does not apply to such cases. Therefore, I would like the authors to clarify whether there exist application scenarios where (1) all tasks are fully known beforehand, yet (2) it is still necessary to learn only one task or a subset of tasks at a time.", "questions_for_authors": "I don’t have other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cB8nh6JytQ", "reviewer": "Reviewer_Z3Hz", "summary": "This paper investigates the task order problem in continual learning. Specifically, it first establishes a theoretical connection between task order and final performance, which is then empirically validated through toy examples. Additionally, the paper examines the impact of task order from two key perspectives: task similarity and typicality. The findings suggest that an optimal task order should adhere to two principles: (1) tasks should be arranged from least representative to most typical, and (2) adjacent tasks should be more dissimilar. Experimental results on small-scale datasets demonstrate the effectiveness of the proposed method.\n\n--------------- Post-Rebuttal Summary ---------------\n========================\nI appreciate the authors' efforts in preparing the rebuttal. While some of my concerns have been addressed, a few still remain. I will maintain my current rating as leaning toward acceptance. However, I strongly encourage the authors to include a discussion on the applicability of task order in real-world scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments. Please find our replies below (we quoted your comments only partially due to space constraint). \n\n> Does this imply that we have access to all tasks when adjusting task order?\n\nWhile we need some information on all tasks, we do not need to have access to all data for adjusting the task order.\nTo clarity this point, we newly implemented an experiment where we estimated the optimal task order from a small subset of training data. \n\n| Data Size  | 0.1\\%         | 1\\%           | 10\\%          | 100\\%         |\n|:--:|:--:|:--:|:--:|:--:|\n| Peri-core  | 60.89 ± 0.94 | 63.30 ± 0.76 | 64.06 ± 0.71 | 64.55 ± 0.71 |\n| Max-path  | 59.75 ± 0.82 | 61.58 ± 0.81 | 62.07 ± 0.78 | 62.22 ± 0.73 |\n| Min-path  | 57.89 ± 0.82 | 56.19 ± 0.67 | 55.80 ± 0.67 | 55.28 ± 0.63 |\n| Core-peri  | 57.60 ± 1.15 | 52.65 ± 1.05 | 52.03 ± 0.97 | 51.02 ± 0.95 |\n| Control | *58.78 ± 0.62*  |   \n\nThe table above shows the average performance of various task orders on split CIFAR-10, where task similarity and the resulting order are estimated from a small subset of data.\nThe \"Data Size\" in the top row indicates the percentage of training data used for the similarity evaluation described in Eq. 16. For example, when Data Size = 1\\%, we used 1\\% of one task’s training data to train a network and evaluated its zero-shot transfer performance using 1\\% of training data from another task.\nThe 100\\% column corresponds to the task setting studied in the manuscript.\nAs shown in the table, even when using only 1\\% of the data, the max-path and periphery-to-core rules consistently outperformed the control, defined as the average performance over random task orders. Note that the control performance doesn't depend on the data size by definition. \n\nThe results above suggest that task-order optimization is feasible even with limited observations, opening the door to many practical applications.\nFor example, consider training a self-driving algorithm across three distinct environments: a suburban area in Phoenix, the city center of Tokyo, and the mountainous regions of the Swiss Alps. Due to geographical and economic constraints, it may be impractical to conduct simultaneous multi-task training across all three locations. While offline multi-task learning is an option, it requires full data acquisition beforehand. An alternative approach is to run small-scale pilot experiments in each environment, measure task similarity, and then determine the most effective order for conducting full-scale training.  Our work offers a theoretical foundation for this approach.\n\nA similar challenge arises in the context of multi-task learning in robotics. In real-world settings, switching between tasks often involves physically rearranging objects around the robot, which is both time-consuming and labor-intensive. As a result, switching tasks on a trial-by-trial basis is often impractical, necessitating block-wise training instead. In such scenarios, optimizing task order could help maximize average performance across all tasks.\n\nWe therefore believe that task-order optimization with partial observations has strong potential for real-world application.\n\n> The evaluations are small-scale with limited comparisons...\n\nThis is indeed an important direction for the future work.\n\n> All analyses are based on linear models and toy tasks (e.g., linear regression), limiting their generalizability.\n\nWe agree that our theoretical results derived in a linear model have limited direct applicability. However, our numerical results indicate that the proposed ordering rules generalize to deep nonlinear neural networks, including convolutional networks and multi-layer perceptrons. \n\n> The methodology behind Figure 2(c) is unclear...\n\nIn this figure, we parameterized three tasks $A$, $B$, and $C$ by their pairwise input correlations $(\\rho_{AB}, \\rho_{BC}, \\rho_{CA})$ and identified the optimal task order for each triplet. For every combination of $(\\rho_{AB}, \\rho_{BC}, \\rho_{CA})$, we computed the error for all six possible task orders using Eq. 8, and assigned a color to the point based on the order that yielded the lowest error. The optimal order was unique except along the boundaries between regions. We sampled $\\rho_{AB}$ and $\\rho_{BC}$ from the range $[0, 0.005, 0.01, \\dots, 0.995]$, and $\\rho_{CA}$ from $[0.2, 0.4, 0.6, 0.8]$, resulting in 160,000 evaluated triplets (which was made possible by our simple analytical expression). The white region corresponds to combinations of $(\\rho_{AB}, \\rho_{BC}, \\rho_{CA})$ that do not constitute a valid correlation matrix. For example, if $\\rho_{AB} = 1.0$ and $\\rho_{BC} = 0.0$, then $\\rho_{CA} = 0.8$ is not a valid value due to the constraints of correlation matrices.\n\n> Some symbols lack explanation, e.g., $\\delta$ in Line 160.\n\nThank you for pointing out. $\\delta_{ik}$ in Line 160 is Kronecker delta, which returns 1 if $i = k$ and returns 0 if $i \\neq k$.", "claims_and_evidence": "All claims are supported by some theoretical or empirical evidences.", "methods_and_evaluation_criteria": "Yes, the proposed method and the adopted evaluation criteria align with the targeted problem and motivations.", "theoretical_claims": "I have generally checked proofs provided in the appendix.", "experimental_designs_or_analyses": "Yes, I have reviewed the experimental designs and analyses in this paper. While I did not find any major issues, I have concerns regarding the experimental details and task scale.", "relation_to_broader_scientific_literature": "This paper have properly discussed with its counterparts.", "essential_references_not_discussed": "No essential references are missed.", "other_strengths_and_weaknesses": "Additional strengths:\n\n - This paper is well-motivated by both theoretical and empirical evidence, and the derived conclusions are clear and insightful.\n\n- This paper is well organized.\n\nAdditional weaknesses:\n\n- Does this imply that we have access to all tasks when adjusting task order? If so, why not use multi-task learning instead of continual learning?\n\n- The evaluations are small-scale with limited comparisons. It remains unclear whether the task order problem is equally important across different CL approaches.\n\n- All analyses are based on linear models and toy tasks (e.g., linear regression), limiting their generalizability.\n\n- The methodology behind Figure 2(c) is unclear. Could the authors provide more experimental details on how this analysis was conducted?\n\n- Some symbols lack explanation, e.g., $\\delta$ in Line 160.", "other_comments_or_suggestions": "Please refer to the **Other Strengths And Weaknesses**.", "questions_for_authors": "Please refer to the **Other Strengths And Weaknesses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XIfCb5Ua7A", "reviewer": "Reviewer_X6mc", "summary": "This paper analyzed the impact of task order and task similarity on continual learning. With a linear teacher-student model, the authors derived an analytical expression that connects task order and task similarity to the learning performance. The theoretical results further derive two principles (the periphery-to-core rule and the max-path rule), validated by extensive experiments.\n\n## Update After Rebuttal\n\nI thank the authors for their rebuttal. It addressed some of my concerns. I think the concerns about the use of task order has been partially addressed, where the proposed insight is applicable to the scenario with a small amount of available data rather than that without any multi-task data. As also mentioned by other reviewers, the real-world applicability remains a challenge. Overall, I appreciate the theoretical efforts and the interesting findings in this paper. I think it's valuable to better understand the continual learning problem. Therefore, I keep my rating toward accepting this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments. Please find our replies below.\n\n> However, I'm not sure the practical benefits of using these insights to improve continual learning\n\nTo illustrate the practical applicability of our work, let us first note that, our method requires only a small number of data samples to estimate the optimal task order.\nWe newly confirmed it by estimating the task similarity and the optimal order using a small subset of training data in our split CIFAR-10 experiment. \n\nTable attached to the reply to Reviewer Z3Hz describes the average performance of various task orders on split CIFAR-10, where task similarity and the resulting order are estimated from a small subset of data. \nWe found that, even when using only 1\\% of the data, the max-path and periphery-to-core rules consistently outperformed the random task order (please see the explanation below the table for details).\n\nThis result suggests that task-order optimization is feasible even with limited observations, opening up many practical applications.\nFor instance, let us consider training a self-driving algorithm across three distinct environments: a suburban area in Phoenix, the city center of Tokyo, and the mountainous regions of the Swiss Alps. Due to geographical and economic constraints, it may be impractical to conduct simultaneous multi-task training across all three locations. While offline multi-task learning is an option, it requires full data acquisition beforehand. An alternative approach is to run small-scale pilot experiments in each environment, measure task similarity, and then determine the most effective order for conducting full-scale training. \nPlease also note that this problem is somewhat different from conventional curriculum learning settings because the goal is to achieve high performance in all three environments. \n\nA similar challenge arises in the context of multi-task learning in robotics. In real-world settings, switching between tasks often involves physically rearranging objects around the robot, which is both time-consuming and labor-intensive. As a result, switching tasks on a trial-by-trial basis is often impractical, necessitating block-wise training instead. In such scenarios, optimizing task order could help maximize average performance across all tasks.\n\nAnother long-term goal is school curriculum design, in which students aim to master diverse subjects while learning them sequentially. Our work may generalize to this problem, given the parallels between artificial and biological neural networks.\n\nWe thus believe there are many potential applications for our theoretical framework. \n\n> Is it possible to extend the theoretical results to other continual learning settings, such as class-incremental learning?\n\nThank you for raising an interesting point. Our latent model formulation is indeed applicable to class-incremental setting, unlike traditional linear model of continual learning, which cannot address class-incremental setting (as different classes do not interfere by definition).\n\nLet us focus on classification problems. In this context, the task-incremental setting corresponds to the case where $C^{\\text{out}}$ is an all-ones matrix (i.e., $\\rho_o = 1$), since all tasks share the same set of output labels. This motivates the analysis of the $\\rho_o = 1$ case presented in Section 4.\n\nIn contrast, class-incremental learning corresponds to the case where $C^{out}$ is the identity matrix (i.e., $\\rho_o = 0$). This reflects the fact that, in the class-incremental setting, tasks do not share any labels, resulting in zero output correlation across tasks.\nAt $\\rho_o = 0$, the dependence of $G_{\\mu\\nu}^+$ and $G_{\\mu\\nu}^-$ on $\\mu$ and $\\nu$ differs from that in the $\\rho_o = 1$ case (see Eq. 57), leading to a different task order preference. In particular, under $\\rho_o = 0$, $G^+\\_{\\mu\\nu}$ is no longer a monotonically decreasing function of $\\mu$ and $\\nu$, and thus the periphery-to-core rule is no longer a reliable heuristic.\nOn the other hand, $G^-_{\\mu\\nu}$ approximately follows the form $\\frac{m}{(2 - m)^2} (1 - m)^{\\nu - \\mu - 1}$, suggesting that, when $m > 0$, tasks that are temporally nearby should be dissimilar in order to minimize error. This is because $\\frac{m}{(2 - m)^2} (1 - m)^{d - 1}$ is a monotonically decreasing function of $d$ when $m > 0$. Therefore, the max-path rule is expected to remain a useful heuristic even in the class-incremental setting.\n\nSince our similarity estimation method is not directly applicable to the class-incremental setting, applying this theoretical framework to nonlinear class-incremental image classification tasks remains a non-trivial challenge. Nonetheless, we believe this is an important direction for future work.\n\n> I think CIFAR-10/100 is still relatively simple and well-crafted, rather than \"real-world\".\n\nWe agree that applying our method to datasets that better reflect real-world scenarios is an important direction for future work.", "claims_and_evidence": "Both theoretical and empirical results are presented clearly and convincing.", "methods_and_evaluation_criteria": "The proposed method is reasonable to me.", "theoretical_claims": "I check the main results, but not all of the details.", "experimental_designs_or_analyses": "The experimental results are extensive and insightful.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper is well organized and easy to follow.\n\n2. The analytic expression is very interesting. It provides a clear connections of task order and task similarity to continual learning, which has been empirically investigated in literature yet without strong theoretical foundations.\n\n3. The theoretical results, especially for the proposed two principles, are validated by extensive experiments. \n\nWeaknesses:\n\nI appreciate most of the contributions in this paper. However, there remains some minor concerns about applicability of the theoretical results.\n\n1. This paper focused on the impact of task order and task similarity on continual learning. It's good to have more in-depth theoretical insights into these critical elements. However, I'm not sure the practical benefits of using these insights to improve continual learning, since the task order in continual learning usually cannot be known and controlled in advance (or else it should be curriculum learning). As a result, the proposed two principles may not provide practical benefits in applications.\n\n2. As acknowledged by the authors, the entire framework is based on task-incremental learning, one of the most basic continual learning settings. Is it possible to extend the theoretical results to other continual learning settings, such as class-incremental learning? Would the impact of task order and task similarity change?\n\n3. The authors claimed that they validate the theoretical results on real-world image classification datasets (Fashion-MNIST, CIFAR-10, CIFAR-100). I think CIFAR-10/100 is still relatively simple and well-crafted, rather than \"real-world\". I'm curious if the proposed principles are applicable to larger-scale images, such as ImageNet, but it's not a big issue.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: [1]From a research perspective, the proposed method is innovative and carries some significance. [Reviewer r3vn]
[2] The paper presents sufficient experimental results with relatively thorough analysis [Reviewer r3vn, Z3Hz]
[3] 	This paper is well-motivated by both theoretical and empirical evidence, and the derived conclusions are clear and insightful [Reviewer Z3Hz, Reviewer X6mc]
[4] The strengths of this paper lie in its rigorous derivations and its novel perspective on continual learning tasks through task ordering. [Reviewer r3vn]

Weakness:
[1] Applicability of task order in real-world scenarios of continual learning is ambiguous [Reviewer r3vn, Reviewer Z3Hz, Reviewer X6mc]
Decision: Accept (poster)