Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper starts by that observing the local minima obtained in a multi task scenario are connected with a linear path of low error regime to the local minima of each task in a continual learning scenario in contrast to the path between the different minima of tasks incrementally learned, provided the both training of multi task and continual learning have started from the same initialization. The paper studies and shows this mode connectivity empirically. It further discusses and analyses the factors behind this connectivity while noting that this is valid when tasks have shared structure in which local minima can be found nearby.
Motivated by these observations, the paper proposes a new solution to the continual learning problem. This is done by defining a new loss that forces this connectivity between the minima of  the previous task and the current task. As this requires evaluating the loss of a previous task, an experience replay of stored previous samples is used.  The paper shows improved performance in comparison to existing methods on different benchmarks of 20 tasks long each.
While I enjoy reading the paper, I think the clarity of the text can be enhanced specifically when referring to figures. The second reference to figure 2, comments on the Euclidean distance without explaining where this is shown in the figure and that was not so clear in the figure caption either. Figure 7, it is not clear what corresponds to the Naïve SGD and what corresponds to the MC SGD.
On the empirical evaluation, I wonder how stable SGD would perform if was given access to the same replay buffer?  It would be also interesting to show the comparison of the path with Stable SGD since it is supposed to find wider local minima where other tasks minima are likely to be nearby.
I assume that split cifar 100 is multi-head, would the proposed solution shows similar advantages in the shared head scenario?

review_title: The paper studies continual learning from the perspective of multi-task learning and shows that a linear path of low error regime connects the found local minima of the subsequent tasks  with that of multi-task learning. 

final_score: rating: 7: Good paper, accept
review_content: ############## Summary ##############

This submission asks the question of how the minima found by batch multi-task learning compare to those of continual learning. It empirically finds the they are connected via linear interpolation through a manifold of low error, and leverages this fact to come up with a clever new algorithm for continual learning that performs better than various existing continual learning baselines on three benchmark data sets.

############## Strengths ##############

1. The question of how to connect multi-task to continual learning solutions is well motivated via simple introductory experiments.
2. The answer to this question, that there is a linear mode connectivity, motivates a simple, elegant, and effective algorithm.

############## Weaknesses ##############

1. The paper could benefit from substantial editing to make it clearer and easier to follow. I found myself having to re-read various sections to properly understand how the different parts of the paper were connected.
2. The empirical evaluation is done only on three benchmarks. It could be valuable to add evaluations on additional data sets, like Omniglot (https://github.com/brendenlake/omniglot).

############## Recommendation ##############

I recommend this paper for acceptance, but urge the authors to substantially revise their manuscript to make it more approachable. I believe this paper to be self-contained, with a clear question being asked, which hadn't been asked before: how are the solutions to multi-task and continual learning methods connected. The authors find that there is linear connectivity between these solutions, and use this fact to motivate a simple yet effective continual learning algorithm.

############## Arguments ##############

The question of whether and how the solutions to multi-task and continual learning are connected is highly relevant. While most prior literature had assumed that some distance metric in the parameter space was the correct way to measure their connection, this work is motivated by the experiments in Fig. 2, which show that these metrics are not quite appropriate. Instead, the authors show that linear mode connectivity better explains how multi-task and continual learning solutions are related.

The manuscript then deviates to an analysis of when this type of connectivity holds by analyzing second-order Taylor approximations. I had to re-read this section (Section 3) multiple times in order to find what the relevance of it was to the submission. My conclusion was that the point is that the fact that the parameter vectors move in directions of low curvature means that interpolation in those directions doesn't increase the loss by much. This fact seems to be somewhat hidden in the text. I encourage the authors to place emphasis on what their analysis is attempting to find before diving into it in depth, as it is easy to lose the reader if they are not aware of where the analysis is going from the start.

The proposed algorithm is clever and simple: it leverages past data not only to approximate the loss of the previous tasks on the new solution, but also to add a regularization encouraging a low-loss linear path between the solutions. Although the authors experiment with very few data sets, I believe they sufficiently show the applicability of their method and the fact that it performs well. It would be interesting to see how differently the method would perform if instead of the MC regularization, the authors used the EWC one. This would help avoid conflating the claim "regularization + replay is best" from "MC regularization + replay is best". Similarly, it would be relevant to reproduce Figure 7 with the solutions found by baselines, to assess whether they also find linear connectivity solutions. The claims would be stronger if the authors showed that baselines don't find linearly connected solutions.



############## Additional feedback ##############

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.


Intro
- It seems like the authors interchangeably used en-dash and em-dash. They also used en-dash to open, but not close, a statement.
- What confounding factors are removed by doing w1 --> w1,w2 other than initialization? The text makes it sound like there's more but no other is discussed.
- Contribution 3: benchmark --> benchmarks
- I believe compressed related work sections or those pushed to the appendix make it hard to place the contribution in context. I encourage the authors to expand this section in the main paper.

Sec 3
- I had two main questions when reading this section:
    - Why doesn't EWC find such a low curvature path, if it precisely penalizes deviations in directions of high curvature?
    - Why can't we just use the proper Taylor expansion instead of Euclidean distance then, to measure forgetting, instead of mode connectivity?
    -These two questions were answered towards the end of the section by showing that this is not a sufficient condition, and are then explicitly addressed by suggesting that second-order approximations are a promising direction for future work. I encourage the authors to clarify this before diving into the analysis, so the reader knows what to look for when reading this section.   
- The caption for Fig. 5 doesn't explain difference between b and c, which is only somewhat explained in text later.

Sec 4
- Regularization only considers low-loss path between the solution to the immediately previous task and the current solution (but not the solutions to all past tasks), assuming that the immediately previous solution contains sufficient information. Was this empirically tested? The EWC authors claim that using only the previous model in their setting is insufficient [1], so it would be interesting to see if there's a similar effect here.

Appendices
- Very complete: additional results, justification of experimental setting.

Style, grammar:
- appendix X --> Appendix X
- second order Taylor --> second-order Taylor expansion/approximation
- minima is often used as a singular, which should be minimum
- regularization based --> regularization-based
- rehearsal based --> rehearsal-based
- Inconsistent italization of i.e.
- few shot learning --> few-shot learning


[1] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hassabis, D. (2018). Reply to Huszár: The elastic weight consolidation penalty is empirically valid. Proceedings of the National Academy of Sciences, 115(11), E2498-E2498.
Chicago 
review_title: The question of how MTL and CL solutions are connected is well motivated, answered satisfactorily, and leveraged to motivate an effective approach to CL

final_score: rating: 7: Good paper, accept
review_content: Summary

The paper studies the relation between the geometry of solutions of continual (CL) and multi-task learning (MTL). Towards this end, the authors empirically identify that all the solutions of CL (i.e. solutions obtained after each task) and MTL are connected by a linear region of low error. This is a very interesting finding and, to my knowledge, has not been studied previously in the CL literature. Based on this observation, the authors propose a memory and regularization-based CL method, MC-SGD, that ensures that the final CL solution is linearly connected to all the task’s solutions. The authors further demonstrate that the solution of the MTL lies in the region where the Hessian of the loss function is low and hence the regularization-based approaches that make use of curvature information (e.g.) EWC, are a promising direction for CL. Experiments are conducted on Permuted and Rotated MNIST, Split CIFAR benchmarks. MC-SGD performs strongly compared to other baselines. 

Positives

1- I quite enjoyed reading the paper. It is very well-written and insightful. 


2- Sections 2.1 and 3 are very nice. The finding, albeit empirical, that the solutions of multi-task and continual learning are linearly connected could prove to be very important for future research in CL. 


3- Experimental results are very strong. I am frankly quite surprised that the gain on top of ER is that much. Although the authors mention it in Section 5 that they use a similar setup as in the other works, I just want to clarify the number of epochs here. If my understanding of their work is correct then Chaudhry et al., in all their work use a single-epoch setup where Farajtabar et al., used multiple epochs. Do you use single or multiple epochs?


Negatives

I don’t have any major concerns about the work except for a few nitpicks and questions. 


1- See the multiple-epochs remark above. 


2- Fig.5: Can you compute all the eigenvectors and show whether the cosine similarity in the eigenvectors corresponding to the smallest eigenvalues actually increase when you go towards the multi-task solution. You can reduce network size if compute is the problem. 

3- Eq.5 (or similarly Eq. 3): It seems that one needs the solution of task t $\hat{w}_t$ for this loss to work. If one just receives the task t how would one obtain this solution? Do you do this in two steps? Where, in step 1, you just compute the $\hat{w_t}$ starting from $\bar{w}_{t-1}$, and then, in step 2, you use the $\hat{w_t}$ obtained from step 1 to compute the final $\bar{w}_{t}$?

4- Fig. 7: Might want to highlight in the legend which path is CL and which is MC. 

5- Page 5, 6th to last line, ~oneself~ itself
review_title: Connection between MTL and CL is very nice!

Meta-review: The paper is presenting an important empirical finding. When the learning algorithms are initialized with the same point, the continual and multitask solutions are connected by linear and low-error paths. Motivated by this finding, the paper proposes a new continual learning algorithm based on path regularization. The paper received unanimously good scores. I agree with the reviews and recommend acceptance. 
Decision: Accept (Poster)