Linear Mode Connectivity in Multitask and Continual Learning
Authors: Seyed Iman Mirzadeh
Mehrdad Farajtabar
Dilan Gorur
Razvan Pascanu
Hassan Ghasemzadeh
Venue: ICLR 2021 Poster (2021)
Keywords: continual learning
catastrophic forgetting
mode connectivity
multitask learning

Abstract: Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to find a solution that performs well on all considered tasks. The main difference is in the training regimes, where continual learning can only have access to one task at a time, which for neural networks typically leads to catastrophic forgetting. That is, the solution found for a subsequent task does not perform well on the previous ones anymore. 
    However, the relationship between the different minima that the two training regimes arrive at is not well understood. What sets them apart? Is there a local structure that could explain the difference in performance achieved by the two different schemes? 
    Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected. We empirically find that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both. We thoroughly analyze this observation and discuss its significance for the continual learning process.
    Furthermore, we exploit this finding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution.  We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks.

Published as a conference paper at ICLR 2021
LINEAR MODE CONNECTIVITY IN MULTITASK AND CONTINUAL LEARNING
Seyed Iman Mirzadeh Washington State University, USA seyedi man mi rzadeh @wsu edu
Mehrdad Farajtabar" DeepMind, USA Farajtabar@goog com
Dilan Gorur
Razvan Pascanu DeepMind, UK
Hassan Ghasemzadeh
DeepMind, USA
Washington State University, USA hassan. ghwsemzadehhwws.. edu
dilang@google.com com
zazp@google.co com
ABSTRACT
Continual (sequential) training and multitask (simultancous) training are often attempting solve the same overall objective: to find a solution that performs well on all considered tasks. The main difference is in the training regimes, where continual learning can only have access to one task networks typically leads to catastrophic forgetting. That is, the solution found for a subsequent task does not perform well on the previous the relationship between the different minima that the two training regimes arrive not well understood. What sets them apart? Is there local structure that could explain the difference in performance achieved by the two different schemes? Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected. We empirically find that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both. We thoroughly analyze this observation and discuss its significance for the continual learning process. Furthermore, we exploit this finding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution. We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks
time, which for neural
anymore. However,
INTRODUCTION
One major consequence of learning multiple tasks continual learning (CL) setting - where tasks are leamned sequentially, and the model can only have access to one task at a time - is catastrophic forgetting (MeCloskey & Cohen 1989). This is contrast to multitask learning (MTL), where the learner has simultaneous access to all tasks, which generally learns to perform well on all tasks without suffering from catastrophic forgetting. This limitation hinders the ability of the model to learn continually and efficiently. Recently, several approaches have been proposed to tackle this problem. They have mostly tried to mitigate catastrophic forgetting by using different approximations of the multitask loss. For example, some regularization methods take quadratic approximation of the loss of previous tasks (e.g. KKirkptriik et al. [2017] Yin et al.) 2020). As another example, rehearsal methods attempt directly compressed past data either by selecting representative subset (e.g. Chaudhry et al. 2019 Titsias et al. 2019) or relying on generative models (e.g. [Shin et al. 2017
Robins 1995).
In this work, we depart from the literature and start from the non-conventional question of under- standing "What is the relationship, potentially in terms of local geometric properties, between the multitask and the continual learning minima?". Our work is inspired by recent work on mode con-
"Equal contribution
'The code is available at http://ccmm/aaaaaee//c-------

---

Published as a conference paper at ICLR 2021
nectivity (Draxler et al. 2018: Garipov et al. 2018 Frankle et al. 2020) finding that different optima obtained by gradient- optimization methods are connected by simple paths of non-incneett increascgg loss. We try understand whether the multitask and continual solutions a manifold of low error, and what is the simplest form this manifold can take. Surprisingly. we find that a linear manifold, illustrated in Fig. Fig. right, reliably connects the multitask solution to the continual ones, granted that the multitask shares same initialization with the continual learning as described below. This is a significant finding in terms of understanding the phenomenon of catastrophic forgetting through the lens of loss landscapes and optimization trajectory and also for designing better continual learning algorithms.
also connected by
To reach this conclusion, we consider particu- lar learning regime described in Fig. [ where after learning the first task using the data D1, we either se sequentially learn a second task obtaining W2 or continue by training on both tasks simul- tancously (i.e., train on D1 + D2), obtaining the multitask solution w We investigate the relation- ship between the two solutions th and w². Note that w is not the typical multitask solution, which would normally start from Wp and train on both datasets. We chose this slightly non-conventional setup to minimize the potential number of con- founding factors that lead discrepancies be-
Wyy
Tisk
D
Task
Takk (Mulitakk)
(Cantimaal)
10g
Depiction of the training regime considered. First úh is learned on task 1. After- wards we either reach W by learning second task w by training on both tasks simultaneously. Right: Depiction of linear connectivity between w2 and úØ and between wq and 22
Figure I: Left:
tween the two solutions (Fort et al. 2019). We also rely on the observation from (Frankle et al. 2020 that initialization can have big impact on the connectivity between the solutions found on the same task, and sharing the same starting point, we do between Ui and w.. might warrant a linear path of low error between the two solutions. Moreover, Neyshabur et al. (2020) noted that in the context of transfer learning, there is performance barrier between two minima that start from pre-trained trained weights, which suggests that the pre-trained weights guide the optimization to a flat basin of the loss landscape. In contrast, barriers clearly exist if these two minima start from randomly initialized weights.
Our contributions can be summarized as follows:
1. To the best of our knowledge, our work is the first study the connectivity between continual learning and multitask learning solutions.
2. We show that compared to conventional similarity measures such as Euclidean distance or Central Kernel Alignment (Korbblihh et al.) 2019), which are incapable of meaningfully relating these minima, the connectivity through manifold of low error can reliably be established. And this connecting path is linear, even when considering more than 20 tasks in a row.
3. Motivated by this, we propose an effective CL algorithm (Mode Connectivity SGD or MC-SGD) that is able outperform several established methods on standard CL benchmarks.
1.1 RELATED WORK
With the trending popularity of deep learning, continual learning has gained a critical importance because the catastrophic forgetting problem imposes key challenges to deploy deep learning models
in various applications (e.g Lange al. 2019 Kemker et al., 2018)) A growing body of research has
attempted to tackle this problem in recent years (e.g Parisi et al. 2018 Toneva et al. 2018 Nguyen
al.
2019 Farajtabar et al. 2019 Hsu et al., [0188] Rusu et al. 2016
Li et al. 2019 Kirkpatrick
et al.
2017 Zenke et 2017 Shin et al. 2017 2017. Rolnick et al. 2018
Lopez-Paz & Ranzato 2017 7
Chaudhry et | 2018b] Riemer et al. 2018 Mirzadeh et al.] 2020] Wallingford et al.] 2020). Among
these works, our proposed MC-SGD bares most similarities rehearsal based methods such us (c.g.
Shin et al. 2017 Chaudhry et al. 2018b) and regularization based methods (c.g. Kirkpatrick et al.,
2017, et al.) (017) similar to Titsiaa et al. 2019). Following (Lange et al. 2019), one can
categorize continual learning methods into three general categories, based on how they approach
dealing with catastrophic forgetting.
Experience replay: Experience replay methods build and store a memory of the knowledge learned so far (Rebuthi et al. 2016 Lopez-Paz & Ranzato 2017 Shin al. 2017 Riemer et al. 2018 Rios & Itti]
2

---

Published as a conference paper at ICLR 2021
2018:
2019). As examples, Averaged Gradient Episodic Memory (A- (A-GEM) Chaudhry 2018b) builds an episodic memory of parameter gradients. while ER-Reservoir Chaudhry al. 2019) uses Reservoir sampling method to maintain the episodic memory.
Zhang et al.
et al.
Regularization: These methods explicitly apply regularization techniques to ensure parameters do not change too much (Kirkpatrick et al. 2017 Zenke al. 2017 Lee al. 2017 (1177,Aljum Aljundi et al. 2018
Kolouri et al.] 2019). They can either a Bayesian (Nguyen et al. 2017]] litsias et al. 2019 |Schwarz
al. 2018 Ebrahimi et al. 2019 Ritter et al. 2018) or frequentist views (Farajtabar et al. 2019 He
& Jaeger 2018 Zeng et al. 2018). For instance, Orthogonal Gradient Descent (OGD) ( Farajtabar
al.)
2019) projects the prediction gradients from new tasks on the subspace of previous tasks'
gradients to preserve the knowledge.
Parameter isolation: Parameter isolation methods allocate different subsets of the parameters to each task (Rusu et al., 2016 Yoon et al.) 2018 Jerfel et al.. 2019 Rao et al. 2019] Li et al. 2019).
From the subility-plasticiie perspective, these methods implement gating mechanisms that improves the stability and controls the plasticity by activating different gates for each task. Masse et al. (2018) proposes a bio-inspired approach for a context-dependent gating that activates non-overlapping subset of parameters for any specific task. Supermask in Superposition (Wortsman et al. 2020) is another parameter isolation method that starts with a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) such that the model achieves good performance. Recently, Mirzadeh et al. (020)) have shown that dropout implicitly creates different pathways or gates for tasks, therefore, reduces their mutual interference and leads to less forgetting.
Continual learning as problem expands beyond dealing with catastrophic forgetting, one of the hopes behind sequential learning is that it can potentially enable positive forward transfer, as one can build on the previously acquired knowledge. In this sense it connects to topics such as Meta- Learning (Beaulieu et al. (2020) Jerfel et al. 2019 He & Jaeger. 2018 Riemer et al. 2018). Few-Shot Learning [Wen et a [2018] Gidaris & Komodakis 2018). Multi-task and Transfer Learning (He & Jaeger, Jerte et al. (019)) It also aims to work in scenarios where task boundaries are not well
defined or provided, or when the data distribution shifts slowly or when multi-task solution does
not exist (Rao et al. 2019: Aljundi et al. 2019 He et al.
2019 Kaplanis et al. 2019).
Mode connectivity (Draxler et al.) 2018 Garipov et al. 2018), is novel tool to understand the loss landscape of deep neural networks. It postulates that different optima obtained by gradient-based optimization methods are connected by simple paths of non-increasing loss (i.e., low-loss valleys). Recently, various works provided different thcoretical explanations for mode connectivity in different scenarios (Venturi et al. | 2019; Kuditipudi et al.) 2019)) either by relying (Neyshabur et al., 2020) investigated the connection between minima obtained by pre-trained models versus freshly intiialized ones. They note that there is no performance barrier between solutions coming from pre-trained models, but there can be barrier between solutions of different randomly initialized models. (Frankle et al.) (2020) shows that different minima that share the same initialization point are connected by a linear path, even with weight pruning. We rely on this observation when designing our setting that the multitask and continual learning minima share a common starting point.
over- parametrization
or concepts such as noise-stability or dropout-stabilie
2 THE RELATION BETWEEN MULTITASK AND CONTINUAL MINIMA
One question driving this work is understanding the relationship between the two different solutions: multitask learning vs. continual learning. In particular, we are interested in scenarios where multitask solution for the considered tasks exists (for a discussion when this does not hold see (He) et al. 2019) and when both learning regimes have the same objective. finding solution that performs well on all tasks. We posit that this difference can not be reliably explained by simple and typical distance metrics used to measure similarity between the trained models. In particular we consider Euclidean distance and Central Kernel Alignment (CKA) (Kornblith et al. 2019). However, these solutions are connected by paths of low error, and, provided that learning starts from the same initial conditions, these paths can have linear form.
In Fig. 2 left column we can see the performance of Naive SGD for multitask and continual learning. Details on the experimental setup can be found in the Appendix The dashed line represents the multitask solution at convergence, which achieves strong performance on all tasks. It further shows the performance of all tasks during the sequential learning experience (each point represents the
3

---

Published as a conference paper at ICLR 2021
Minima
861"
0.0
57
12.6
7.5
21.6
0.95
0.97
0.59
0:96
577
0.0
10.5
5.7
19:8
0.50
0.98
8/2
40g Minima
602
Tasks Learned
layers
Mg layers
(a) Rot-MNIST forgetting
(b) Rot-MNIST Euclid. distance (c) Rot-MNIST CKA distance
1.5
Accurasy
t g
Minime
0/ blooks
0.90
1.00
60
thy
9.5
0.91
0.93
0.92
0.97
-05
4
Tasks Learned
Minima
blocks
blocks
(d) Split CIFAR forgetting
(e) Split CIFAR Euclid. distance
(f) Split Cifar CKA distance
Figure 2: Continual and Multitask learning performance and relation between minima. Top row:
Rotation MNIST. Bottom row: Split CIFAR-I00. Left column: Validation accuracy of all tasks during continual training. Middle: Euclidean distance: note that úhs is not good solution for task 1 even though it is closer (more similar) ii than w in Euclidean distance. Right: CKA distance: i is closer (more similar) to ú than 2g in terms of CKA distance. Therefore, neither Euclidean nor CKA distance able to realize MTL. is better in avoiding catastrophic forgetting.
performance after learning another task), highlighting how performance on past tasks degrades considerably. Note that as deseribed in Fig. and further detailed in Appendix (.2 Fig. [5] in the multitask learning scenario, tasks are added sequentially to the loss, to construct a parallel with the continual learning setting. This will be the case throughout this work.
Eucledian distance. It might be reasonable to expect that the less the parameter changes, the less the forgetting will be. One can motivate this heuristic on Taylor expansion of the loss, as done in (Mirzadeh et al. 2020), where, the forgetting is defined as:
L (102) - L1 (w)) = 7 i2 - w1) V2 Li (w)) - w1) I
X1
max | - w111
(I)
Here, (w) is the empirical loss for task 1, and XqTix is the largest eigenvalue of its Hessian at Φ Note that all terms of the Taylor expansion are multiplied with úب wi and hence the norm of this delta will affect the amount of forgetfulness. In fact, this is frequently done when pruning neural networks (Zhu & Guptal 2018 | Han et al.) 2015), where weights are zeroed out based on magnitude, producing minimal Euclidean distance to unprunned model. But, observed in Fig. 2 (middle column), Eucledian distance does not correlate with not suffering from catastrophic forgetting: the CL solution on task (w² is closer to the CL solution of task (t0;) than the multitask solution of tasks 5 (wg). One explanation could be that the bound defined by Eq. (1) will not be tight if the vector úis- thi does not lie in the direction of the largest eigenvector. Appendix D.I contains further details on this topic.
Centered Kernel Alignment. Centered Kernel Alignment (CKA) (Korbblith al. 2019) measures the similarity of two representations on the same set of examples. Given N examples and two xd2
activation outputs on these examples, Ri E Nxd and R2 Nxd2 CKA is defined by:
|RR R 12
CKA(RI, Rd) ||R] R. R R2
(2)
where, ||- is the Frobenius norm. Recent work by Ramasesh et al. 2020 studies catastrophic forgetting on the CIFAR dataset by measuring the CKA similarity score of different layers of ú and
W2. They argue that the later layers suffer more from catastrophic forgetting by showing that the CKA similarity initial layers decreases less after training on sequential tasks.
However, the CKA score suffers from few shortcomings. If the number of training epochs per task is small (c.g., in streaming case), the CKA does not change much, even though the accuracy for previous tasks drops drastically. For instance, in Fig. 2 rright column, we show that the pairwise CKA between different layers of the first task minimum (w ) and CL and multitask minima of task 2 and task 5 are roughly the same. Although, the phenomenon observed in (Ramasesh al. 2020) is still realizable by a very tiny margin. Moreover, we can see that the multitask minimum of task 2 (w²), and task 5 (wx) are more similar ú compared CL minima (ıº and úr))
4

---

Published as a conference paper at ICLR 2021
2.1
MODE CONNECTIVITY
Mode connectivity has been studied empirically (Draxler et al., 2018 Garipov et al. 2018) and theoretically with some assumptions (e.g., over-vvevattteeee paramettttttnnnoonnn or noise stability) (Kuditipudi et al.) 2019 Venturi et al. 2019h. We note that these previous works were focused on "single-task" setup, where the data stays i.i.d., and more importantly, the networks start from different initialization. Because of this assumption, it is shown that the minima are not linearly connected (i.e., the loss is not necessarily decreasing in the interpolation line between the two minima). Instead, the path consists of several sengeessents or curves (Kuditipudi et al. 2019).
However, between our continual and multitask settings, a more favorable property can exist, namely that learning starts from the same initialization (i.e., W1 wf). For "single task", common initialization leads to linear path between minima (Frankle et al. 2020). Therefore, our first step is to investigate whether the same linear connectivity holds between multitask and CL solutions. As we show in Fig. 3] this is true for both MNIST and CIFAR benchmarks. In this figure, we can see that the multitask minima for task (w?) is connected to both task I CL minima (ú²) and task 2 CL minima (162), We note that the surface plots in Fig. Brre not of low-dimensional projection types and are rather "interpolation plots" in parameter space computed on the hyper-plane ppare-plane connecting these three minima. More details are provided in Appendix
C.5
>168
2.
2225
1b
Wilsccoon Loo (Takk
1a
Willooo Las [Takk
27
22
19
Mitacca Loo
19
17
17
12
10
13
(a) MNIST Task loss
(b) MNIST Task
loss
(c) CIFAR Task
loss
(d) CIFAR Task loss
Figure 3: Cross-entropy validation loss surface of on rotation MNIST (a and b), and split CIFAR-100
(c and d), as a function of weights in a two-dimensional subspace passing through w1, 12 and w²-
Pulls
Falls
Pethi
looe
Tox
0.00
0.21
11.50
0..5
1.00
0.00
0.25
0.90
0.25
1.00
6,00
0.25
01.44
0.75
1.50
0.00
11.25
11:00
6.75
1.00
Inderpoltiin
Interpelation
Inserpclatio
Interpo
(a) MNIST wi to next ones (b) MNIST in to next ones (c) CIFAR wh to next ones (d) CIFAR th to next ones
Figure 4: Exploring the loss along the linear paths connecting the different solutions: The loss increases the interpolation line between the first task solution(ú]) ( and subsequent continual solutions, while the loss remains low on the interpolation line between 0ū and subsequent multitask minima (a and c). The same observation also holds for the second task solution (182) (b and d)
To demonstrate linear connectivity holds for subsequent MTL and CL minima, we plot the validation loss on the line- segment interpolating between all subsequent minima for both task and 2 on Rotated MNIST and CIFAR in Fig. [ The low loss interpolation to wn indicates that indeed along this linear path, the loss stays low.
3 WHEN AND WHERE DOES LINEAR MODE CONNECTIVITY HOLD?
The fact that linear regions of low error connect MTL and CL minima does not necessarily imply it is trivial to find one such path. In this section, we try to analyze this finding through the geometry of loss functions and, more specifically. its curvature, which identifies the amount of change when move from solution of task I (wi ) to either MTL (w;) or CL (w²) solution. This way, we can explain why existing methods such EWC that are also trying to penalize the change in directions of high curvature fail to find a plausible direction. The short answer is that they only rely on the second-order Taylor approximation of the loss function, which may be easily violated. We elaborate as follows.
Let us start by trying to understand what is implied by the fact that these linear paths exist. Focusing on the linear path between úi and w one potential justification is that while minimizing the multitask
5

---

Published as a conference paper at ICLR 2021
loss, the updates move in the direction of low curvature for the Hessian of the first loss around 21. To see why, let us make the mild assumption that the function is smooth, which typically tends to hold in the over- over-parameterized regime (e.g., similar assumptions are needed for the NTK regime to hold (Bietti & Mairal 2019)). Then we can not have a rapid change in the curvature. Hence, if moving along the line connecting Wh and wzz the loss of task 1 stays very small, it means the curvature also has to be small as it can not show rapid fluctuations that will not be reflected in the loss. Implicitly the curvature is low ú6 as well; hence this direction necessarily has to be within the subspace spanned by the eigenvectors with low curvature. This seems not to hold for un and 22 so we might expect that the line connecting them to move in directions of higher curvature. A symmetric argument can be made for w2 and W2 looking now at the Hessian and loss of task 2. The smoothness argument makes unlikely that curvature shows rapid fluctuations; hence loss staying low implies the alignment with eigenvectors of low curvature.
Before moving forward, let us empirically validate the argument in the previous paragraph. To do so, we estimate the top 50 eigenvalueseeigeneetoo of the Hessian matrix around task minima of rotation MNIST benchmark, similar to the setting we discussed in Section 2 We use the deflated
power iteration method to obtain the eigenspectrum of the Hessian. Fig. ba shows the largest eigenvalues. We note that similar to the observation by FFor & Ganguli (2019) and Mirzadeh et al. (2020), the eigenspectrum of the Hessian includes bulk plus C. larger outlier eigenvalues where C is the number of classes. To quantify the confinement between the subspace spanned by the Hessian eigenvectors and the movement directions, we measure the cosine angle of these directions and the Hessian eigenvectors. Fig. [bb] shows that the direction from ú0 to di lies within the subspace spanned by the top eigenvectors. Note that this agrees with our hypothesis. However, the direction between 2i to w2 does not lie within this subspace as the cosine angle between this direction and the eigenvectors are all near zero, suggesting that this direction is orthogonal to all these eigenvalues and hence is spanned by eigenvectors with low eigenvalues. The orthogonality has also been tried as a hard constraint in a number of continual learning papers explicitly (Farajtabar et al., 2019 He & Jaeger 2017). This explains the less suffering from catastrophic forgetting as we move in directions of low curvature.
Semlatty
Coirr Smilaby
Eigendla
9.4
m.
022 83
0.5
20
30 Eigenvectors (largstt - smallest)
20
Eieevvlue Index (longest smallest)
Eigenvectors (baggst smallest)
(a)
(b)
(c)
Figure 5: Comparison of the eigenspectrum of the Hessian matrix for wh (a): top Eigenvalues. (b and c): The overlap between Hessian eigenvectors and directions from vi W2 and wa within the suitable region that 2nd order Taylor approximation holds (b) and does not hold (c).
Now, let us examine whether moving in the direction of low curvature is sufficient in addition to being a necessary condition or not? Fig. 5c shows that this is not sufficient. By increasing the number of epochs from 5 to 20 we show that both directions tu and w do not lie in this subspace of the top eigenvectors. That is, for both, if we move on the segment away from W1, we are aligned with none of the directions associated with high curvature. But úi suffers from catastrophic forgetting and hence a considerable increase in the loss of task 1, while ws does not. This suggests that while initially moving towards Wy the curvature stays low, but eventually higher-order terms increase the curvature of task loss before reaching dंz Overall this implies the multiple wt that we observe, which gradually solve more and more tasks, lie within region where a second-order Taylor approximation of the first task still holds, and higher-order derivatives do not play an important role. And within this region, learning is restricted to the subspace spanned only by eigenvectors corresponding to small eigenvalues for all previous tasks.
We empirically show that we can learn up to 20 complex tasks and up to 50 permuted MNIST tasks in this regime (see Fig. 7 here and 20 in Appendix D.5). hence it seems this region, where the second-orde -oddrr Taylor approximation holds, is quite expressive and sufficient to learn many tasks without interference between them. Additionally, similar linearizations seem to hold reliably in over parametrized models throughout learning as shown by the NTK-based literature (e.gJacot et al.; 2018)) Bietti & Mairal 2019) and these assumptions are implicitly made by several successful continual
6

---

Published as a conference paper at ICLR 2021
learning algorithms (e.g. Kirkpatrick et al. 2017 Farajtabar et al. 2019 Mirzadeh al. 2020). In particular, this observation is interesting when viewed from the perspective of regularization based CL methods. Many of these methods rely on second-order Taylor expansion (if not of previous losses then of the KI. term with respect to how much the model changed) and forbid change in weights with high curvature. The fact that multitask learning adheres the same principles, moves in the direction of low curvature and restricts itself to the region where this approximation holds, suggests that this remains a promising direction. also highlights the importance of better approximations of curvature and the importance of remaining within the area where this approximation holds.
In Appendix B, we further investigate different aspects of our setting that might make it possible to find reliable solutions for all subsequent tasks in an area around the solution of the first task where the second-order Taylor approximation holds. We show, that as assumed earlier, sharing the same initial condition is vital for linear connectivity to occur. Furthermore, the shared structure between tasks like lewelevel features in images or semantically consistent labeling plays an important role as well. If destroy any structure between tasks, we can not find multitask solution for task I and 2 that is linearly connected to the first task.
CONTINUAL LEARNING WITH MODE CONNECTIVITY
Based on the observation in Section 2. that there exist minima with linear mode connectivity to previous CL minima, in this section, we propose continual learning algorithm that exploits this property. The proposed algorithm can be viewed as a combination of both replay and regularization methods in continual learning.
4.1 MC-SGD
Mode Connectivity SGD (MC-SGD) is light-weight method that constrains the continual minima to lie in low-loss valley to all previous minima. For simplicity, start by providing the loss function for two tasks, then we extend it for an arbitrary number of tasks. We define the loss objective that enforces linear connectivity to both wi and W2 below:
i arg min Josas1 Jose (10 a(w ih1) + C2(122 + a(w (u2))) do,
where L1 and Ci are the loss function for task I and task respectively and parameterizes the line connecting w to ii and ii respectively in the first and second term. Essentially, Eq. 43) constrains the MC-SGD minima to have low-loss path to both wn and dhs. The integral in this equation be approximated by averaging the loss over a few random in [0, 1]. However, our experiments showed that picking as few n 5 equally spaced points between 0 and I is more than sufficient get good results. We can further decompose
into:
arg min 1
(w) + L2(w) + Li (0)) + (2(122)
constant
(10 aaw w1)) L2(w2 + a(w -(2))] (4)
ac
regularization
It's evident from Eq. 4 that T should minimize both task and task 2 loss, in addition to having low-loss paths to both of the CL minima.
In continual learning setting, however, would not have access L1 once start learning Lz. Therefore we rely on experience replay like ER (Chaudhry et al. (219)) and AGEM (Chaudhry small replay buffer with randomly sampled examples to approximate Li,
let al. 2018b). We with the hope that given the restricted form of Fq. (4) through the additional regularizer, we might need lesser data then if would have used it approximate directly the multitask loss.
We can further extend the mode connectivity loss to an online version that is cheap to compute:
T = arg min
ElEn + a(w w;-1)) + ( + a(w -wx))]
(5)
10
Eq. (5) be viewed as follows: when learning task I, assuming that have terh-performance solution for the previous t tasks (i.c., w;-1), we try to minimize Eq. ( as if in this equation, di 1 and the is á.. The replay memory still consists of samples from all the previous tasks.
7

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS AND RESULTS
The experimental setup, such as benchmarks, network architectures, continual learning setting (e.g., number of tasks, episodic memory size, and training epochs per task), hyper- hyper-parameters parameters, and evaluation metries are chosen to be similar several other studies (Chaudhry et al.) 2018b Mirzadeh
et al. 2020 Chaudhry et al. 2019 Farajtabar et al., 2019; Chaudhry et 20199) For all experiments, we report the average and standard deviation over five runs with different random seeds. Appendix C provides a detailed discussion on our experimental setup explained below.
Benchmarks. We report three standard continual learning benchmarks: Permuted MNIST (Good fellow et al. 2013), Rotated MNIST, and Split CIFAR-100. Although we are aware of the short- comings of Permuted MNIST (Farquhar & Gal 2018), for the sake of consistency with literature. we report the result on this benchmark. However, we also report our results on Rotated MNIST and CIFAR- 100, that are more challenging and realistic datasets for continual learning, especially when the number of tasks is large. Each task of permuted MNIST is generated by random shuffling of the pixels of images such that the permutation remains the same for the images of within the same task, but changes across different tasks. Rotated MNIST is generated by the continual rotation of the MNIST images where each task applies a fixed random image rotation (between 0 and 180%). Split CIFAR is variant of CIFAR- 100 where each task contains the data from 5 random classes.
Architectures. For the rotation and permutation MNIST benchmarks, we use a two-layer MLP with 256 ReLU units in each layer. For the split CIFAR- -00 benchmark, we use ResNet18, with three times fewer feature maps across all layers.
Evaluation. Following several studies in the literature (Chaudhry et al.) 2018b; Mirzadeh et al. 2020], we report two metrics to compare CL algorithms when the number of tasks is large. (I) Average Accuracy: The average validation accuracy after the model has been continually learned task is defined by A, = Ei= dr,i, where, de is the validation accuracy on dataset i after the model finished learning task L.
(2) Average Forgetting: The average forgetting measures the backward transfer in continual learning. This metric is calculated by the difference of the peak accuracy and ending accuracy of each task, after the continual learning experience has finished. For a continual learning benchmark with 7 tasks,
it is defined by F = T-1 Ei=
max (I) (as.s - or:)).
5.1 COMPARISON WITH OTHER METHODS
In this experiment, we compare MC-SGD with various established continual learning algorithms. Our setup consists of 20 tasks for three benchmarks: permuted MNIST, Rotated MNIST, and split CIFAR-100. The episodic memory size for A-GEM, ER-Reservoir, and MC-SGD is limited to be one example per class per task. Table [ compares the average accuracy and average forgetting (i.e., A, and F defined above) for each method once the continual learning experience is finished (i.e., after learning task 20). Moreover, Fig. 6 shows the evolution of average accuracy for each method on each benchmark. We can see in the figure that MC-SGD outperforms other methods with its performance gap increasing as the number of tasks increases. In Appendix D.51 we show this trend also holds when the number tasks increases from 20 to 50 for Permuted MNIST benchmark. Overall, the results validate a few aspects of our method. MC-SGD seems make more efficient of the episodic memory compared to replay based techniques A-GEM and ER-Reservoir, perhaps due to incorporating the additional knowledge of linear connectivity as a regularizer. Moreover, if we see our approach as restricting learning in the subspace of low curvature, this subspace is better approximated compared to EWC, which takes an explicit Taylor approximation of the KL term and adds a similar constraint (that learning can not change weights with high curvature).
Table 1: Comparison between the proposed method (MC-SGD) and other baselines.
Permuted MNIST
Rotated MNIST
Split CIEAR-100
Method
Accuracy
Forgelling
Accuracy
Forgelling
Accuracy
Forgelting
Naive SGD
(20.03) (E2.46))
46.3(21.3))
0.52 (20.01)
40.4
0.31 (20.00)
EWC (Kirkpatrick al. 2017
70.7 (111.44
0.23 (10.00)
(...4)
0.48 (10.01)
(21.81
0.28 (110.03)
A A-GEM (Chandhyy et all 2018h
65.7 (40.51)
(+-0.0)
(5.37)
0.42 (4-0.01)
50.7 (+2.32)
0.19 (+-0.4))
ER-Reservoir (Chuudary
2019
(200.44
(10.01)
(9.10=
0.21 (+0.01)
46.9 (=0.76)
0.21 (+0.00)
Stable SGD (Mizzadch all 2020
80.1 (+:0.51)
0.09 (=0.01)
70.8 (=0.88
0.10 (2:000)
(=1.1))
0.08 (#: 0.03)
MC-SGD (ours)
85.3 (110.61)
0.06 (110011)
82,3 (:2688
0.08 (10001)
63.3 12.21)
0.06 (1 0.03)
Multtask Learnin
(9.5(1)
0.0
89.80 10,37)
0.0
68.N (0.72)
0.0
8

---

Published as a conference paper at ICLR 2021
1001
Accuracy
50
Accuracy
St
Accuracy
70 F360
625
2658
10
15
15
20
16
30
Tasks
Tasks
Tasks
(a) Permuted MNIST
(b) Rotated MNIST
(c) Split CIFAR-100
Figure 6: Evolution of average accuracy during the continual learning experience with 20 tasks.
5.2 MODE CONNECTIVITY OF THE MC-SGD MINIMA
Next, we show that by minimizing Eq. 5 the minima found by MC-SGD are almost linearly connected to both the starting point of learning a new task (which represents the multitask solution of all previous task) as well as the solution of the task being learned. Fig. showss the interpolation plots for rotated MNIST and split CIFAR-100, In Fig. 7a we show the validation loss for task on the interpolation line between ún and four subsequent MC-SGD, and continual minima obtained during the learning of all subsequent tasks. We choose Vi and mi as minima the early stages of learning, w10 and 1o for the middle stages of learning, and 115. Wi5. 122. and 22 for the late stages of learning in our illustrations. We can see the losses on the interpolation lines between CL and MC minima are nearly flat compared the losses on the lines among CL minima. Moreover, Fig. 7b shows the interpolation plot for task to make sure the conclusion holds for later minima as well. show the interpolation plots of split CIFAR- 100 for task 1 and 5.
Similarly, Figs. 7c and 7
1.0
(a) MNIST Task
(b) MNIST Task
(c) CIFAR Task
(d) CIFAR Task 5
Figure 7: Mode Connectivity between the CL minima found by Naive SGD and minima found by the proposed MC-SGD: (a) and (b) Rotated MNIST (c) and (d) Split CIFAR- 100. In each figure, the left legend column corresponds paths between Naive SGD minima, and the right column corresponds to paths between MC-SGD minima.
CONCLUSION
While both continual and multitask learning aim to find solutions that perform well across all tasks, the difference in their training regimes leads to two very different outcomes: multitask learning typically ends up achieving this goal, while continual learning suffers from catastrophic forgetting, performing well only on the recently seen tasks. In this work we investigated the relationship between these two solutions when all other potential confounding factors are minimized.
We considered simple training regime in which multitask learning and continual learning start from similar conditions when incorporating new task. We showed that this condition, multitask minima are connected to continual learning ones by a linear path of low error, while continual learning solutions are not similarly connected. This can be understood through the glance of Taylor expansions as multitask objective restricts learning in the directions of low curvature within an area where second order approximation holds. Such solutions seem exists even when the process is applied repeatedly, solving more than 20 tasks in row.
Finally, we explored this observation and proposed new algorithm for continual learning called Mode Connectivity SGD (MC-SGD). It relies on the assumption that there always exist solution able to solve all seen tasks so far that is connected by a linear path of low loss. MC-SGD utilizes a replay buffer to approximate the loss of for previous tasks. However, compared to other popular rehearsal-bart methods, it performs better with less data as it exploits the linear path to further constrain learning.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
SIM and HG acknowledge support from the United States National Science Foundation through grant ENS-1750679 The authors thank Jonathan Schwarz, and Behnam Neyshabur for their valuable comments and feedback.
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 339-154, 2018.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Online continual learning with no task boundaries. arXiv preprint arXiv:190300867. 2019.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.09577, 2020.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems, pp. 12994, 2019.
Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Rie- mannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018a.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. ArXiv, ahss/1812.000 2018b.
Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip H. S. Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. ArXiv, abs/2002.08166, 2019.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: /902. 10486, 2019.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In ICML 2018: Thirty-fifth International Conference on Machine Learning, pp. 1308-1311 2018.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. arXiv preprint 190.002,00., 2019.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. ArXiv, abss/1910.07 00400,0000000 2019.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. ArXiv, abs/1805.09933, 2018
Stanislav Fort and S. Ganguli. Emergent properties of the local geometry of neural loss landscapes. ArXiv, Abss/191,,0592 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv: 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML 2020: 37th International Conference Machine Learning, 2020.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling dnns. NeurIPS 2018: The 32nd Annual Conference Neural Information Processing Systems, pp. 8789-8798, 2018
10

---

Published as a conference paper at ICLR 2021
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367-4375, 2018.
lan J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. An empirical investigation catastrophic forgeting in gradient-basee neural networks. CoRR, abs/ abs/1112.6211, 2013.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. CoRR, http/155600022222 2015. URL http:://blpp..... unittiierr de/ http///anaaaaa/ooooorrrraa.....a
html #Hanl
Xu He and Herbert Jaeger. Overcoming catastrophic interference by conceptors. arXiv preprint arXiv: 707.04853. 2017.
Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor aided backpropa-
gation. In ICLR 2018, 2018.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Tch, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv: 05201, 2019.
Yen-Chang Hsu, Yen-Cheng Liu, and Zsolt Kira. Re-evaluating continual learning scenarios: A
categorization and case for strong baselines. arXiv preprint arXiv: /2488, 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Ghassen Jerfel, Erin Grant, Thomas L. Griffiths, and Katherine A. Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In NeurlPS, 2019.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. arXiv preprint
2019.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. In Thirty-second AAAI conference artificial intelligence, 2018
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, and et. al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 13:3521-3522 2017.
Soheil Kolouri, Nicholas Ketz, Xinyun Zou, Jeffrey Krichmar, and Praveen Pilly. Attention- struttual-pllastccin arXiv preprint arXiv:1903.0607 2019.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML 2019: Thirly-sixth International Conference on Machine Learning, pp. 3519-3529, 2019.
Rohith Kuditipudi, Xiang Wang. Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity low-cost solutions for multilayer nets. In NewrIPS 2019: : Thirty-third Conference on Neural Information Processing Systems, pp. 14574-14583, 2019.
Matthias Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ale Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. ArXiv, abs/1100.00833. 2019.
Sang-Wor Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Neural information processing 2017.
systems, pp.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. arXiv preprint 1rXiv:/904.000 2019.
11

---

Published as a conference paper at ICLR 2021
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems. pp. 6467-6476, 2017.
Nicolas Y. Masse, Gregory D. Grant, and David J. Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences of the United States of America, 115 44, 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. 1989.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, and Hassan Ghasemzadeh. Dropout as an implicit gating mechanism for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 232-233, 2020.
Scyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. arXiv preprint arXiv:2006.006 06958, 2020.
Behnam Neyshabur, H. Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? ArXiv, abs/2008. 1687, 2020.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv: /710. 10628, 2017.
Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv:1908.0100 01091, 2019.
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. 2018.
Adam Paszke, Sam Gross, Francisco Massa, and et. al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019.
Vinay V. Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint S.XXi:2007.074000 2020.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell, Continual unsupervised representation learning, In Advances in Neural Information Processing Systems, pp. 7645-7655, 2019.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542, 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: /810. 1/910, 2018
Amanda Rios and Laurent Itti. arXiv:1811.01 140.1116.661 146, 2018
Closed-loop gan for continual learning.
arXiv preprint
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp. 7738-3748, 2018.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning. arXiv preprint arXiv: 1682, 2018.
12

---

Published as a conference paper at ICLR 2021
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.044 04671, 2016.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Crabska-Barwinsker Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. International Conference on Machine Learning, pp. 4535 4544, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning using gaussian processes. arXiv preprint arXiv: 1901. 2019.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint 1 2018.
Luca Venturi, Afonso Bandeira, and Joan Bruna. Spurious valleys one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, 20(133): 2019.
Matthew Wallingford, Aditya Kusupati, Keivan Alizadeh-Vant Vahid, Aaron Walsman, Aniruddha Kembhavi, and Ali Farhadi. In the wild: From ml models to pragmatic ml systems. ArXiv, fos/2000.025199 2020.
Junfeng Wen, Yanshuai Cao, and Ruitong Huang. Few-shot self reminder to overcome catastrophic forgetting. arXiv preprint arXiv:1812.00543, 2018.
Mitchell Wortsman, V. Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Yosinski, and Ali Farhadi. Supermasks in superposition. ArXiv, abs/2006. 14769, 2020.
Dong Yin, Mehrdad Farajtabar, and Ang Li. SOLA: Continual learning with second-order loss
approximation. arXiv preprint arXiv:2006. 10974, 2020.
Jachong Yoon, Eunho Yang, Jungtae Lee, and Sung Ju Hwang, Lifelong learning with dynamically expandable networks. In Sixth International Conference on Learning Representations. ICLR, 2018.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continuous learning of context-dependent processing neural networks. arXiv preprint 1rXiv:1810.0022 1810.01256, 2018
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 3987- 3995. JMLR, 2017.
Mengmi Zhang, Tao Wang, Joo Hwee Lim, and Jiashi Feng. Prototype reminding for continual learning. arXiv preprint 1905 09447, 2019.
Michael Zhu and Suyog Gupta. To Prune, or Not to Prune: Exploring the Efficacy of Pruning for
Model Compression In International Conference Learning Representations Workshop, 2018.
13

---

Published as a conference paper at ICLR 2021
A OUTLINE
The appendix is organized as follows:
Appendix provides detailed discussion on required assumptions for mode connectivity.
Appendix gives comprehensive discussion on the experimental setup for each experiment, hyper- parameters, visualization, and implementation. addition, we provide anonymous links the code and details to reproduce the results.
Appendix D contains additional experiments and results, including detailed plots for mode connec- tivity of continual and multitask minima in Sec. 2 and additional experiment permuted MNIST dataset with 50 tasks.
B BREAKING THE LINEAR CONNECTIVITY BETWEEN THE CONTINUAL LEARNING AND MULTITASK SOLUTION
B.1 DIFFERENT INTTIALIZATION
Furthermore, useful to understand we can break the linear path between the multitask and continual learning minima. An initial assumption that this is caused by the sharing of the same initialization, where both 2ب and wi are learned starting from ú Fig. shshws how violating this assumption breaks the linear mode connectivity for the split CIFAR-1OU 100 benchmark. Fig. 8a shows the validation loss interpolation between wj and the the next four multitask and CL minima. As illustrated, the multitask minima is no longer linearly connected ú and is separated by a wall. The same phenomenon holds for the next minima, and in Fig. 8c we show this for W2. We note that for MNIST datasets, as reported by Frankle al. (2020) the linear connectivity still holds even if the MTL minimum starts from other initialization points. Related to this, Neyshabur et al. (2020) investigated the connection between minima obtained by pre-trained models versus freshly initialized ones. They pointed out that there is no performance barrier between solutions coming from pre-trained models, but there can be a barrier between solutions of different randomly initialized models. Here, 1 acts like pre-trained model for multitask objective, placing the optimizer in good basin for task 1.
Pale
Vasdane Lam (Takk
TE
0.00
0.25
11.50
0.70
Intopo/litio
Inioruslaiio
(a) 00 to next minima
(b) Task
loss plane
(c) D2 to next minima
(d) Task loss plane
Figure 8: The linear connectivity does not hold for MTL minima that start from different initializations on CIFAR-100
B.2 SHARED STRUCTURE OR SEMANTICS OF DATA
Another potentially important role might be played the shared structure between tasks. If the different tasks do not share common structure it might be unlikely that there exists a solution for one task in the neighborhood of the other. manipulate the shared structure of tasks, we can either corrupt input images corrupt the labels. To increase the distribution shift across tasks, in this experiment, we use a continual learning setup with two tasks: The first task is the MNIST dataset, and the second task is the Fashion-MNIST dataset.
To corrupt the input data, we add different amounts of independent Gaussian noise to the input images of the second task (i.e, Fashion MNIST) and report the results in Fig. 9 Fig. 99 to 9d show the gradual separation of wi and W; as the amount of noise increases. The validation loss values on the linear interpolation between these two minima are also shown in Fig. 9a
14

---

Published as conference paper at ICLR 2021
Fig. 10b to 10d show the effect of randomly changing the labels for different portions of the images of task 2. As illustrated, the more labels we comrupt, the more disconnected th and wy become. The loss values the linear interpolation path between these minima shown in Fig. T0al for different percentage label corruption.
Roes
Gisiim Tiae 20
Lox (Tekk 2)
VVa Dalaaiion amm (TTaskc
Vi Silaan (Taacc
2P
1.25
In
0.5
Interpotation (M2 o5)
(a)
(b) Noise: (0.5, 1)
(c) Noise: (1,
(d) Noise: (11.55, 1)
Figure Noise Injection: (a): The interpolation on task for different amount of Gaussian noise on images, (b), (c), and (d): wz becomes disconnected from W2 the amount of noise increases
17.00
Wasder lom
Valdetion Lown
Valdiion Lons
10
(a)
(b) No label corruption
(c) 5% label corruption
(d) 10% label corruption
Figure 10: Label Corruption: (a): The interpolation on task for different amount of label corruption, (b), (c), and (d): w becomes disconnected from u1 the more labels are corrupted
Yet a very trivial case of breaking mode connectivity is to cut a few classes from second task. To this end, we perform the following experiment: After training on the first task and obtaining ú for the multitask training on the second task, we randomly choose classes out the 10 classes and remove the first task's 's examples that belong any of these classes from the training dataset of multitask model.
In Fig. 1la show that by removing different number of classes from the training of multitask model, the linear mode connectivity does not hold anymore as the validation loss on task 1 increases on the line between Wi and w². Moreover, while without removing any classes we have linear mode connectivity as illustrated in Fig. TTb by removing 2 classes, W2 is no longer connected to ú (Fig. Te.. The trend continues if remove more classes (i.e, 4 classes in Fig. lTd.
Clasa Ramoval
15
Valiacioon Loso 1)
10
Vyyiaatio Look
0.0
6.5
10
Interpolation (6) eg)
(a)
(b) Keep all classes
(c) Remove 2 classes
(d) Remove 4 classes
Figure 11: Removing Classes: (a): Interpolation, (b), (c), and (d): wi becomes more disconnected
remove more and more classes
We emphasize that for common continual learning benchmarks, the assumption of linear mode connectivity holds, and this section aims to push the linear mode connectivity its boundaries in order to appropriately scope this interesting phenomenon that has significant consequences to understanding and development of continual and multitask learning.
15

---

Published as a conference paper at ICLR 2021
C DETAILS OF EXPERIMENTAL SETUP
In this section, we answer two important questions regarding our experiments and result:
1. What decisions did make for experiments (e.g., continual learning setup, model architectures, benchmarks, etc.)?
2. Why did we make those decisions and how close setup is to other established continual learning works?
To this end, we start with the first question by reviewing our experimental setup, followed by hyper- parameters we used in the main experiments. To answer the second question, in each sub-section, show that our setup is close to several established works in the continual learning literature.
C.I BASELINES AND ARCHITECTURES
We chose the baselines and architectures based on three criteria.
The first criterion is the diversity of methods. In addition to the naive SGD, we have selected A-GEM (Chaudhry et al. 2018b) and ER-Reservoir (Chaudhry et al. 2019) as representative of episodic memory methods, EWC (Kirkpatrick et al.] (2017) ) for regularization methods. The second criterion is reproducibility. While there are several other continual learning methods, we only reported the methods that we could replicate their results either by their official implementations or self-implemennatioon.. The final selection criterion is performance. A-GEM and ER-Reservoir are gegh-performance episodic memory methods. In addition, we have added the stable- (Mirzadeh itt al., 2020) as the current state the art continual learning method, in addition to the previous state of the art methods such A-GEM and ER-Reservoir.
At the end of this section, we show that by carefully choosing a similar continual learning setup to common setup in the literature, have made easier to compare our results with other works that we could not replicate ourselves.
C.2 CONTINUAL LEARNING SETUP
In the MNIST and CIFAR experiment in Section 2 we have chosen tasks so that we can show microscopic metrics (c.g., the evolution of accuracy for all tasks) thorough the continual learning experience. In this experiment, our goal was not to reduce catastrophic forgetting. contrast, that experiment, we have aggravated the catastrophic forgetting to show that CKA and Euclidean distance are not helpful, even if the amount of forgetting is high. More specifically, we have not tuned the hyper-parameters, and we increased the catastrophic forgetting by using per task rotation of 22.5 in the rotation MNIST benchmark. Moreover, for MNSIT experiment, we have used a two-layer MLP with 100 ReLU units. The ResNet18 in that experiment is similar to the ResNet18 in Section 5 Finally, in this setup, we have used the learning rate of 0.1 for MNIST dataset and 0.05 for the CIFAR experiment with batch size of 64. Finally, in this experiment, the number of of epochs per task is 5. We discuss the rationale behind this setup in Appendix C
In the comparison experiment in section 5.1 we have increased the number of tasks to 20 tasks, and the model is trained on each task for one epoch. This benchmark is more challenging since we used one example per class per task (i.e.. 200 for rotated and permuted MNIST, and 100 for split CIFAR-100) Thus, the model is allowed to see the data only once, and hence, the setting becomes closer to the real-world online setting. For the CIFAR experiment, to make sure the first task solution is within a good basin, we train the only first task for five epochs in this experiment and continue the training for one epoch for the next tasks. While our results might seem initially lower than the results reported in other continual learning works, we emphasize that this is due to the limited episodic memory we have used in our setup. There are two reasons behind this choice of memory size and number of epochs. First, to make it closer to common setup in other related works such as A-GEM and Reservsvv that are state of the art in episodic memory methods. Second, the limited episodic memory and number of epochs allow measuring "how effective each continual learning method uses the previously seen data". this experiment, we use a two-layer MLP with 256 ReLU units for MNIST datasets and ResNet18 with three times less feature for CIFAR dataset. We note that have
16

---

Published as a conference paper at ICLR 2021
similar setup with our baselines, the task identifiers used to select the correct output head only in CIFAR-100 experiments.
C.3
For the experiment in Section 5.1 we have used the following grid for each model. We note that for other algorithms (e.g., A- A-GEM, and EWC), we ensured that our grid contains the optimal values that the original papers reported.
NAIVE SGD
learning rate: [0.25, 0.1, 0.01 (MNIST, CIFAR-100), 0.001]
batch size: 10
EWC
learning rate: [0.25, 0.1 (MNIST, CIFAR-100), 0.01, 0.001]
batch size: [10, 64]
À (regularization); [1, 10 (MNIST, CIFAR-100), 100]
A-GEM
learning rate: [0.1, 0.1 (MNIST), 0.01 (CIFAR-100), 0.001]
batch size: [10, 64]
ER- RESERVVO
learning rate: [0.25, 0.1 (MNIST), 0.01 (CIFAR-100), 0.001]
batch size: [10, 64]
STABLE SGD
initial learning rate: [0.25, 0.1 (MNIST, CIFAR-10O), 0.01, 0.001]
learning rate decay: [0.9, 0.85, 0.8CCIAR--100), 0.6(MNIST))
batch size: [10, 64]
dropout: [0.25 (MNIST), 0.1 (CIFAR-100))
MODE CONNECTIVITY SGD
To obtain continual minima (i.e., un to (220), we use the same hyper-parameters as the stable SGD. To minimize the objective function in Eq. (3) and obtain wj to U2 we use the following grid:
number of samples: [3. 5. 10. 20). We found that empirically more than five samples on each line would give sufficient information to minimize Eq. for both MNIST and CIFAR experiments.
learning rate: [0.001, 0.01 (CIFAR-100), 0.05(MI 0.1, 0.2 (MNIST, CIFAR-100). 100)].
17

---

Published as a conference paper at ICLR 2021
C.4 How CLOSE OUR EXPERIMENTAL SETUP TO THE OTHER WORKS IN THE LITERATURE?
We end this section by showing that setup is close to the common setup in the continual learning literature. More specifically, we discuss the following aspects of our setup:
Benchmarks: While believe continual learning literature can benefit from more diverse set of tasks such as reinforcement learning tasks, the majority of the focus in the field is devoted to computer vision tasks. One possible explanation is that despite having experience, and despite the fact that is possible to train high-pes gigh-performance models in single task settings, still, models perform relatively poorly in continual learning setting. Hence, we followed the common benchmarks in the literature such as rotated and permuted MNIST, and split CIFAR- 100. While we believe the community needs agree on more diverse of benchmarks, the performance gap between continual learning minima and multitask minima is yet to be filled even on these simpler benchmarks.
control over the continual learning
The choice of two-layer MLP with 256 hidden neurons for MNIST benchmarks and & for CIFAR- 100 benchmark is common in the literature (Chaudhry et al.) 2018a] Mirzadeh al. 2020, Chaudhry al. (011)). We have used same architectures to make it easier to compare our results with other works as well.
Architectures:
C.5 IMPLEMENTATION DETAILS
In this section we discuss the visualization and implementation details.
C.5.1 How ARE SURFACE PLANES GENERATED?
To plot a surface plane (e.g., , Fig. 3 we need three points to obtain two basis vectors. Suppose we have set of three parameters W1, W2, and wa where each parameter is in vector in high-dimensional space. These vectors can be obtained by computing the flattened version of each layer the neural network and then concatenating them. For the three mentioned points, perform the following procedure:
1. Calculate two basis vectors: ã = u2 - W1, and T = W1.
2. Orthogonalize the basis vectors by calculating e U cos v)u
3. Define Cartesian coordinate system in the (x.y) plane that maps each coordinate parameter space by calculating p(z, y) = w1 + x v
4. Finally, for defined grid on this coordinate system, calculate the empirical loss of each (x, y) coordinate using function p(r, y).
Finally, we note that to generate plots that show loss along the interpolation between minima (e.g.. Fig. 4) we simply calculate the interpolation solution between two minima and calculate the loss for that solution. For example, the minima on the interpolation line that connects vi to th2 can be represented with Wa = in + a - w1) where 0 I a S We then calculate the empirical loss for Wa-
C.6 REPRODUCING MC-SGD
We provide executable version of MC-SGD on the CodaLab platform. CodaLab keeps track of the full provenance generating a particular result, and thus, executable papers on this platform allow others to verify the results independently.
To this end, we provide five different runs of MC-SGD on rotation MNIST with 20 tasks to verify the results in Table I Each uses different random seed and can be fully replicated it has all the required dependencies and the source code. Here are links to run run run 3) run 4, and funn 5)
Each of the above links has "stdout" section that logs the evolution of the average accuracy of MC-SGD. addition, each run generates a "checkpoint" folder that includes weight files to MC-SGD weight files at the end of each task.
18

---

Published as a conference paper at ICLR 2021
C.6.1 MC-SGD IMPLEMENTATION
To implement MC-SGD, we use PyTorch Passkee al. 2019) because it dynamic graph capability,
The challenging part of the implementation is calculating Eq. ( where we need to compute summation of gradients on lines. To this end, we first calculate the accumulated gradients over the mentioned line:
def
on interpolation line (w_ hat, w bar, alphas, memory)
Get accumulated gradients interpolation line between w_hat and w_bar
accumllated_gracientsss line 0.0
for each point on the interpolation line for alpha alphas:
interpolation []
(1) caicuiate the parameters on the interpolation line interpolation. w_hat + (w_bar w_hat) alpha model
latroo_prra_eeettee.e veettrriinteppllaiion.. loss calculate_loss (model, memcry)
(2) calculate the loss to obtain gradients loss backward New the gradients are stored in the graph
(3) collect the gradicnts
for param in model. paremeterrmmmerssseee ():
get gradients for each modulle/block/layer
(param, maram..grdd.. vied (-1))
= torch. cat (Interpollttion_grada)
(4) accumuiate gradients
accumulated gradientsdaaeeeee line interpolation
return accumlltee__rradients.oo...
D ADDITIONAL RESULTS
D.1 CONVENTIONAL DISTANCE MEASURES
In this section, we provide detailed lz distance and pair-wise CKA scores the models introduced
in Sec. 2
As explained in Sec. [ l2 distance can be misleading metric when it comes to comparing the multitask solution to the continual learning one. Le.. the continual solution task 5 seems be closer to the solution of task 1 than the multitask solution, even though the continual solution suffers from catastrophic forgetting. This could be indicative the fact that compared to the pruning scenario, curvature might play bigger role in continual learning and can not be ignored (e.g., the delta between parameters might align differently to the eigenvectors the Hessian and of higher order derivatives).
However, the bound defined by Eq. (I) is not accurate enough the vector 22 ú does not lie in the direction of the largest eigenvector. For instance, it might be the case (as we show later) that the optimization trajectory moves in a direction with low curvature (e.g., a low-loss valley), and thus the loss does not increase. The ineffectiveness of Euclidean distance is also verified empirically by middle column Fig. 2 Even though the MTL minima for tasks are further from CL minima they have much higher accuracy. For the reference and further evidence, the full table of pairwise distances between all minima is shown Fig. 12
D.2 ALL MINIMA ARE LINEARLY CONNECTED
19

---

Published as a conference paper at ICLR 2021
0.0
5.7
8.8
11.0
12.8
7.5
12.7
17.2
21.6
20
8
4.2
6.8
6.1
10.0
M-2
00
4.2
0.0
3.7
5.3
6.0
4.9
6.3
7.8
9.3
5.1
0.0
5.0
8.1
7.8
10.1
14.4
18.9
15
W13
5.6
3.7
0.0
3.9
4.7
6.1
6.8
9.5
7.5
18
11.0
8.2
5.0
0.0
5.0
10.2
11.5
14.5
18.7
404
5.3
8.0
2.7
7
10.0
12.8
10.5
8.1
5.0
0.0
12.1
13.3
15.7
19.1
10
7.4
6.0
4.7
2.7
0.0
7.7
8.3
9.1
10.1
550
5.7
10.2
12.1
0.0
7.8
13.2
18.1
121
1.2
3.9
7.8
12.7
10:4
10.1
11.5
13.3
7.8
0.0
8.4
14.1
-
6.1
6.3
7.8
8.3
3.9
0.0
6.1
2.5
17.2
2
14.4
14.5
15.7
13.2
8.4
0.0
9.0
M4
7.7
7.8
81
8.7
9.1
5.9
4.0
0.0
4.2
0.0
W2
10g
Wg
M4
wg
Parameters
Parameters
(a) Rotated MNIST
(b) Split CIFAR-100
Figure 12: Pairwise & distances between all CL and MTL minima. For each CL minima, while MTL minima are further, they still are good minima due to the fact that they lie in a low-loss valleys, we show in section 2.1
1.00
1.00
6:44
2:48
3:47
layer
3.99
0:40
40
00
400
0.00
0.0
413
Tayer
0:00
1:22
0.7
2:00
3:55
Kaper 2
2:03
0.00
Laper
Loyet
1ype
kyyer
(a) CKA(W1,,A)))
(b) CKA(K), .5))
(c) CKA($1,1ss)
(d) CKA(W
Figure 13: Rotated MNIST: Pairwise CKA similarity score between hidden layers
bock
1:00
E.50
0100
Blcck
1.00
5:51
0.02
block
2.99
bloc
0:03
01:00
8:10
3.90
0:80
2:54
0.00
2.5
0.04
5.53
0.60
3.90
1U5
3:65
1:00
6:40
3.97
400
bock
0/
0.
block
0.44
5:49
0.40
block
3.99
book
0.45
6:44
3.53
0.30
0.04
0.03
5.03
0.67
2:30
Mock
3:62
0.96
0:67
3.97
Blcck
blcck
(a) CKA(W1, .
(b) CKA(@1...))
(c) CKA(61.ú5))
(d) CKA(11
Figure 14: Split CIFAR-100: Pairwise CKA similarity score between ResNet blocks
In this section, first show the extended learning setup for 3 tasks in Fig. 15 Escentially, this is the extended version Fig where an additional task added.
Wig
P
10
wi
Moreover, provide more detailed result for the interpolation plots in Fig. where we show zoomed version of those plots. For easier comparison, we
Task (Contiaual)
+
Task 2 (Mullitask)
+
Task 3 (Contimunl)
Tisk 3 (Multitukk
repeat the plots from Fig. ]here in Fig. 16b
16d
17b
and 17d with the same order. Fig 16a shows two
paths (i.e., Wh W2 and wi w:) from Fig.
16b
Figure 15: Setup for continual and multitask learning with tasks
Although the path from wi to úh (in yellow) seems nearly non-increasing Fig. 16b in Fig [6a we show that this path in fact an increasing path while the loss on the interpolation path between úi and thy is non increasing. Similarly, this phenomenon is shown in Fig. 16c for th to dy and wg on rotated MNIST dataset and in Fig. for split CIFAR- 1-0 dataset.
20

---

Published as a conference paper at ICLR 2021
0.1
Poth kg
Tlax L00
022
11.00 Incpulation
0.35
100
0,00
0.35
0.50
0.75
1.00
Intrpslation
Irtespdlatin
Intepolicinn
(a) 10 to 20 and ws
(b) Wi to next minima
(c) 202 to tis and ug
(d) W2 to next minima
Figure 16: Detailed loss plots for interpolation paths between minima on rotated MNIST
Petr 45
Texc Lon
100
0.00
01.00
1.00
Intryslation
Irterpolation
Iroepolicion
(a) 11 to 22 and ws
(b) t0 to next ones
(c) 22 to d0g and us
(d) 12 to next minima
Figure 17: Detailed loss plots for interpolation paths between minima on split CIFAR-100
D.3 STABLE SGD METHOD WITH MEMORY
One interesting question to pursue would be how would the stable SGD method perform if given same size of episodic memory. To this end, we add an episodic memory to the stable-SGD method with similar size of MC-SGD (i.e., one example per class per task).
We show the performance of the stable SGD method in Table on Rotated MNIST and Permuted MNIST with 20 tasks, with a similar setup in Sec. [.1 We can see in Table 2 the additional episodic memory improves the average accuracy by 2.1% in Permuted MNIST and 2.5% in Rotated MNIST.
Table 2: Performance of the stable SGD method with access to episodic memory
Permuted MNIST
Rotated MNIST
Method
Average Accuracy (%)
Forgetting
Average Accuracy (%)
Forgetting
Stable SGD
80.1 0.51)
0.09 (= 0.01)
70.8 0.78)
0.10 (#: 0.02)
Stable SGD
82.2 (# 0.83
0.09 (= 0.01)
73.3 (+ 0.97)
0.10 (+ 0.03)
episodic memory
we compare the loss on the interpolation paths between SGD, Stable-SGD with memory, and SGD methods. To show Stable SGD minima, use w notation. We can that the additional episodic memory helped the stable-SGD method to gain a significant boost. Still, the regularization term in the MC-SGD method helps it find minima with better mode connectivity.
Moreover, Fig. 18
Paths
Loss
SGD
S-SGD Mom
MC.SGD
kily
Task 1-
19 PPo
s) viz0
0.0
0.2
0.4 Interpolation
0.6
0.8
1.0
Figure 18: Interpolation plots for SGD, stable SGD with memory, and MC-SGD Rotation MNIST benchmark with 20 tasks: Stable SGD with memory cannot find linearly connected minima.
21

---

Published as a conference paper at ICLR 2021
D.4 MODE CONNECTIVITY OF EWC MINIMA
One interesting question that if regularization methods such as EWC can implicitly help obtain minima that linearly connected. Similar the Fig. and Fig- [8] we show the interpolation paths for EWC minima In Fig. [9 We can see that although the EWC minima perform better than SGD, they are linearly connected to the first task's minimizer.
Paths
Task 1 Loss 2
SGD
MC:SGD
W8 - 48g
ui -7 8is + us
181
88
tho
ui -7 800
681
8b
ui -7 803
0.2
0.8
Interpolation
Figure 19: Interpolation plots for SGD, EWC, and MC-SGD on Rotation MNIST benchmark with 20 tasks: the EWC method cannot find linearly connected minima.
D.5 PERMUTED MNIST WITH 50 TASKS
In this experiment, the MLP hidden layers increased from 256 512 for this experiment since the number of tasks increased. Moreover, the size of episodic memory is increased from 200 to 500 (one example per class per task).
100
Mode Connectivity
Stable SGD
Accuracy
ER- FR-Reesrvvirr
EWC
A-GEM
10
40
Tasks
Figure 20: Evolution of the average accuracy for different methods on permuted MNIST with 50 tasks.
22


Reviews:
[{"review_id": "i5ViUokelNP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper starts by that observing the local minima obtained in a multi task scenario are connected with a linear path of low error regime to the local minima of each task in a continual learning scenario in contrast to the path between the different minima of tasks incrementally learned, provided the both training of multi task and continual learning have started from the same initialization. The paper studies and shows this mode connectivity empirically. It further discusses and analyses the factors behind this connectivity while noting that this is valid when tasks have shared structure in which local minima can be found nearby.\nMotivated by these observations, the paper proposes a new solution to the continual learning problem. This is done by defining a new loss that forces this connectivity between the minima of  the previous task and the current task. As this requires evaluating the loss of a previous task, an experience replay of stored previous samples is used.  The paper shows improved performance in comparison to existing methods on different benchmarks of 20 tasks long each.\nWhile I enjoy reading the paper, I think the clarity of the text can be enhanced specifically when referring to figures. The second reference to figure 2, comments on the Euclidean distance without explaining where this is shown in the figure and that was not so clear in the figure caption either. Figure 7, it is not clear what corresponds to the Naïve SGD and what corresponds to the MC SGD.\nOn the empirical evaluation, I wonder how stable SGD would perform if was given access to the same replay buffer?  It would be also interesting to show the comparison of the path with Stable SGD since it is supposed to find wider local minima where other tasks minima are likely to be nearby.\nI assume that split cifar 100 is multi-head, would the proposed solution shows similar advantages in the shared head scenario?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ND0YD0w5F5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper studies the relation between the geometry of solutions of continual (CL) and multi-task learning (MTL). Towards this end, the authors empirically identify that all the solutions of CL (i.e. solutions obtained after each task) and MTL are connected by a linear region of low error. This is a very interesting finding and, to my knowledge, has not been studied previously in the CL literature. Based on this observation, the authors propose a memory and regularization-based CL method, MC-SGD, that ensures that the final CL solution is linearly connected to all the task’s solutions. The authors further demonstrate that the solution of the MTL lies in the region where the Hessian of the loss function is low and hence the regularization-based approaches that make use of curvature information (e.g.) EWC, are a promising direction for CL. Experiments are conducted on Permuted and Rotated MNIST, Split CIFAR benchmarks. MC-SGD performs strongly compared to other baselines. \n\nPositives\n\n1- I quite enjoyed reading the paper. It is very well-written and insightful. \n\n\n2- Sections 2.1 and 3 are very nice. The finding, albeit empirical, that the solutions of multi-task and continual learning are linearly connected could prove to be very important for future research in CL. \n\n\n3- Experimental results are very strong. I am frankly quite surprised that the gain on top of ER is that much. Although the authors mention it in Section 5 that they use a similar setup as in the other works, I just want to clarify the number of epochs here. If my understanding of their work is correct then Chaudhry et al., in all their work use a single-epoch setup where Farajtabar et al., used multiple epochs. Do you use single or multiple epochs?\n\n\nNegatives\n\nI don’t have any major concerns about the work except for a few nitpicks and questions. \n\n\n1- See the multiple-epochs remark above. \n\n\n2- Fig.5: Can you compute all the eigenvectors and show whether the cosine similarity in the eigenvectors corresponding to the smallest eigenvalues actually increase when you go towards the multi-task solution. You can reduce network size if compute is the problem. \n\n3- Eq.5 (or similarly Eq. 3): It seems that one needs the solution of task t $\\hat{w}_t$ for this loss to work. If one just receives the task t how would one obtain this solution? Do you do this in two steps? Where, in step 1, you just compute the $\\hat{w_t}$ starting from $\\bar{w}_{t-1}$, and then, in step 2, you use the $\\hat{w_t}$ obtained from step 1 to compute the final $\\bar{w}_{t}$?\n\n4- Fig. 7: Might want to highlight in the legend which path is CL and which is MC. \n\n5- Page 5, 6th to last line, ~oneself~ itself", "extra_scores": {}, "extra_text": {}}, {"review_id": "6J2slrBwTuW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############## Summary ##############\n\nThis submission asks the question of how the minima found by batch multi-task learning compare to those of continual learning. It empirically finds the they are connected via linear interpolation through a manifold of low error, and leverages this fact to come up with a clever new algorithm for continual learning that performs better than various existing continual learning baselines on three benchmark data sets.\n\n############## Strengths ##############\n\n1. The question of how to connect multi-task to continual learning solutions is well motivated via simple introductory experiments.\n2. The answer to this question, that there is a linear mode connectivity, motivates a simple, elegant, and effective algorithm.\n\n############## Weaknesses ##############\n\n1. The paper could benefit from substantial editing to make it clearer and easier to follow. I found myself having to re-read various sections to properly understand how the different parts of the paper were connected.\n2. The empirical evaluation is done only on three benchmarks. It could be valuable to add evaluations on additional data sets, like Omniglot (https://github.com/brendenlake/omniglot).\n\n############## Recommendation ##############\n\nI recommend this paper for acceptance, but urge the authors to substantially revise their manuscript to make it more approachable. I believe this paper to be self-contained, with a clear question being asked, which hadn't been asked before: how are the solutions to multi-task and continual learning methods connected. The authors find that there is linear connectivity between these solutions, and use this fact to motivate a simple yet effective continual learning algorithm.\n\n############## Arguments ##############\n\nThe question of whether and how the solutions to multi-task and continual learning are connected is highly relevant. While most prior literature had assumed that some distance metric in the parameter space was the correct way to measure their connection, this work is motivated by the experiments in Fig. 2, which show that these metrics are not quite appropriate. Instead, the authors show that linear mode connectivity better explains how multi-task and continual learning solutions are related.\n\nThe manuscript then deviates to an analysis of when this type of connectivity holds by analyzing second-order Taylor approximations. I had to re-read this section (Section 3) multiple times in order to find what the relevance of it was to the submission. My conclusion was that the point is that the fact that the parameter vectors move in directions of low curvature means that interpolation in those directions doesn't increase the loss by much. This fact seems to be somewhat hidden in the text. I encourage the authors to place emphasis on what their analysis is attempting to find before diving into it in depth, as it is easy to lose the reader if they are not aware of where the analysis is going from the start.\n\nThe proposed algorithm is clever and simple: it leverages past data not only to approximate the loss of the previous tasks on the new solution, but also to add a regularization encouraging a low-loss linear path between the solutions. Although the authors experiment with very few data sets, I believe they sufficiently show the applicability of their method and the fact that it performs well. It would be interesting to see how differently the method would perform if instead of the MC regularization, the authors used the EWC one. This would help avoid conflating the claim \"regularization + replay is best\" from \"MC regularization + replay is best\". Similarly, it would be relevant to reproduce Figure 7 with the solutions found by baselines, to assess whether they also find linear connectivity solutions. The claims would be stronger if the authors showed that baselines don't find linearly connected solutions.\n\n\n\n############## Additional feedback ##############\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\n\nIntro\n- It seems like the authors interchangeably used en-dash and em-dash. They also used en-dash to open, but not close, a statement.\n- What confounding factors are removed by doing w1 --> w1,w2 other than initialization? The text makes it sound like there's more but no other is discussed.\n- Contribution 3: benchmark --> benchmarks\n- I believe compressed related work sections or those pushed to the appendix make it hard to place the contribution in context. I encourage the authors to expand this section in the main paper.\n\nSec 3\n- I had two main questions when reading this section:\n    - Why doesn't EWC find such a low curvature path, if it precisely penalizes deviations in directions of high curvature?\n    - Why can't we just use the proper Taylor expansion instead of Euclidean distance then, to measure forgetting, instead of mode connectivity?\n    -These two questions were answered towards the end of the section by showing that this is not a sufficient condition, and are then explicitly addressed by suggesting that second-order approximations are a promising direction for future work. I encourage the authors to clarify this before diving into the analysis, so the reader knows what to look for when reading this section.   \n- The caption for Fig. 5 doesn't explain difference between b and c, which is only somewhat explained in text later.\n\nSec 4\n- Regularization only considers low-loss path between the solution to the immediately previous task and the current solution (but not the solutions to all past tasks), assuming that the immediately previous solution contains sufficient information. Was this empirically tested? The EWC authors claim that using only the previous model in their setting is insufficient [1], so it would be interesting to see if there's a similar effect here.\n\nAppendices\n- Very complete: additional results, justification of experimental setting.\n\nStyle, grammar:\n- appendix X --> Appendix X\n- second order Taylor --> second-order Taylor expansion/approximation\n- minima is often used as a singular, which should be minimum\n- regularization based --> regularization-based\n- rehearsal based --> rehearsal-based\n- Inconsistent italization of i.e.\n- few shot learning --> few-shot learning\n\n\n[1] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hassabis, D. (2018). Reply to Huszár: The elastic weight consolidation penalty is empirically valid. Proceedings of the National Academy of Sciences, 115(11), E2498-E2498.\nChicago ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is presenting an important empirical finding. When the learning algorithms are initialized with the same point, the continual and multitask solutions are connected by linear and low-error paths. Motivated by this finding, the paper proposes a new continual learning algorithm based on path regularization. The paper received unanimously good scores. I agree with the reviews and recommend acceptance. 
Decision: Accept (Poster)