Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new heuristic approach to continual learning in which a set of models ("modes") are joinlty updated after each task, as in an ensemble method, but using a novel loss function, which encourages diversity. Some empirical gains (relative to some other heuristic methods) on some task incremental learning image classification benchmarks are shown.
strength_and_weaknesses: The idea of using a mixture of sub-models ("modes"),, instead of a single large parameter model, seems intuitively reasonable, although is not very novel (there is a large prior literature on ensembles and mixture of experts, etc). Furthermore, the paper makes claims that I think are false, and is very hard to read.

One of the biggest problems is that the authors claim they do not need to know about task boundaries. Yet it is clear from algorothm 1 that the data is actually provided to the algorithm in chunks, with each chunk corresponding to a different task, and the algorithm is allowed to exploit this fact , by storing different parameter vectors for each mode i and task t, in the form theta(i,t). This seems contradictory. Furthermore, it seems the algorithm stores multiple copies of theta(i,t), one per epoch, to get theta(i,t,e). Thus the running time in step 12 is O(t E N) for each task t, resulting in a total complexity of O(T^2 E N). This seems expensive in time and space - more so than a simple replay buffer strategy (which they do not compare to).
Another problem is that their predictions are computed by using an unweighted average of the logits from each mode. What is the justification for this? If the modes are specialized on different tasks, why should they all get to vote equally?

The presentation is very unclear. For example, what does thm 1 even mean? theta_d^MTL is never defined. And why should we care about how much paramters drift or not? What does this have to do with predictive performance? Also, what assumptions are you making on the data stream (data distribution for each tasks)? 

Many terms are either not explained (eg "EWC" is not defined), or very poorly explained. For example, the distinction between instance-IL, task-IL and class-IL is very unclear. It seems to have something to do with the label (output) space for each task, Y_t ( some partitioning of a fixed, larger ("fine grained") label set. ) . It seems the algorithm assumes ahead of time that every task will always be a C-way classification task, but the "meaning" of each label may change across tasks (eg for task 1, 0=dog, 1=cat, 2=horse; for task 2, 0=truck, 1=car, 2=airplane), which we can think of as a hierarchical two-part label, (task, class). So it seems once again that you do need to know the task boundaries.

The experiments are not very clear. 
- In sec 4.0, What does it mean to say "We utilize resnets-{18,50,152} with {11, 181, 642 || 23, 528, 522 || 58, 164, 298} adjustable parametrs? What does the || notation mean? What are these adjustable parameters, and which ones are fixed? 
- In sec 4.1, you compare to MTL as your "gold standard", but what exactly is this? (I assume you train a single big model with T output heads, each with C classes, in an offline way?) 
- In sec 4.2, the meaning of fig 3 is unclear. It seems that the loss does not vary at all in the subspace of  mode 2, suggesting that a single mode may suffice. Also, for mode 1, the low loss region seems constant across tasks, but this might be an artefact of your regularizaton strategy. Once again, why should we care about how the parameter subspaces change? What does this have to do with predictive performance?
- In table 3, you only compare to somewhat older baseline methods (<=2018) -are these SOTA? And you only compare on 1 dataset (split cifar-100). Are the differenes between methods significant? (No error bars are reported!) Can the diferences be explained away by hyper-parameter tuning?
- tables 1-3 are all hard to read (bolding the best and second best might help, and/or focus on fewer metrics).




clarity,_quality,_novelty_and_reproducibility: Clarity is very low - see above.

Quality seems low - only compares to one prior method (EWC), which is quite old and probably not SOTA.

Novelty seems small -there are several prior works on online learning with ensembles and sparse models.

Reproducibility seems reasonable.
summary_of_the_review: The paper proposes some ad-hoc heuristics which are poorly explained and seem lacking in theoretical justification. The experimental results seem inconclusive.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a novel method, Mode-Optimized Task Allocation (MOTA), for continual learning with superior backward and forward transfer performance as well as lower average task drift.

The paper first identifies the key weakness behind prior regularization-based continual learning methods: they anchor all future parameters to the first task observed, even when doing so degrades transfer performance. MOTA instead aims to leverage the global geometry of the parameter/loss space by instead optimizing $N$ different modes, each having a set of tasks allocated to it. It proves that when $N$ and the set of tasks per mode are optimized against parameter capacity $|\theta|$, the total task drift is lower than when $N=1$.

It describes the MOTA algorithm which first initializes the $N$ modes such that the pairwise distance among them are maximized, and updates the parameters for each task via checkpointed gradient updates such that the drift per mode is minimized.

The paper concludes after presenting empirical comparison of the transfer performance of MOTA with other baseline continual learning methods, which shows that its average accuracy is superior to all except multi-task learning.
strength_and_weaknesses: Strengths:
- Provides clear motivation to study multi-mode continual learning (Theorem 1).
- Shows creativity in the construction of its algorithm.
- Solid experimental result and its discussion.

Weaknesses:
- Theoretical justification for the specific steps of its algorithm (Section 3) lacking.
- In particular, motivations behind (1) the choice of parameter prior that maximizes the pairwise distance and (2) using random-weight-interpolated parameter in its loss function are unclear.
- The second paragraph of Section 3.2 (starting with "If a task has a high level of certainty, ...") is confusingly written and may benefit from an illustrative diagram or a figure.
- The paper claims that the algorithm does not depend on the knowledge of the task boundaries, and yet the specific implementation of the algorithm seems to depend on it (e.g. regularizing distance with parameter of previous task). Further clarification would be helpful.
- Conclusion is abrupt and does not provide any direction for future research.
clarity,_quality,_novelty_and_reproducibility: The paper is, for the most part, very clear and easy to understand. It seems to be highly original, though "we are amongst the first to leverage the global geometry of the loss landscape for task adaptation," seems to suggest there may be other works with similar insights, and a more detailed comparison and claims of novelty would have been appreciated.
summary_of_the_review: Overall, the paper clearly identifies a limitation with prior regularization-based continual learning methods and suggests an alternative that is both theoretically and empirically well-supported and convincing. Although parts of its main algorithm lack clear motivation and its conclusion does not offer insights about its potential impact and directions for future research, it is an original and creative work that I strongly recommend.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider the usage of more than one models for CL problem, and apply two regularization terms to maximize the difference of all models at 1st task and to minimize the parameter drift.

The paper claims better model quality when compared to other method.
strength_and_weaknesses: Strength
1) Two regularization are proposed. One to max the difference of all models at 1st task. One to reduce param space drift between two tasks.
2) The MOTA work should be easy to enable and user friendly for many real world tasks.

Question
1) The two regularization both use the weighted factor when added to the loss. How robust would the model metric be when the weighted factor change?
2) As the results in Table 3 have base line approach from papers before or in 2018, are they the best base line model for now, or should the comparison add newer approach from papers after or in 2019?
clarity,_quality,_novelty_and_reproducibility: Clarity and quality are very nice.

Novelty seems mostly on the practical aspect.
summary_of_the_review: Recommend accept, marginally over threshold. Major reasons for not higher score would include the extent for novelty and baseline model selection in the comparison, whether more advanced and newer methods are compared to MOTA.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
The paper studies continual learning with multiple modes. The proposed method MOTA is motivated by mode connectivity and ensembles literature. MOTA aims to train $N$ diverse solutions on the first task, while for the next tasks, it uses the average/joint prediction to minimize the training loss and parameter drift. In addition, at the end of training each task, MOTA uses stored checkpoints to further optimize the drift. The authors show the effectiveness of MOTA on various tasks and instance incremental scenarios.



strength_and_weaknesses: ### Strengths


- The paper is easy to follow, and the method is intuitive. 


- The idea of continual learning with multiple modes and the trade-off analysis is interesting. 



### Weakness:


- First, I should note that in terms of novelty and contributions, continual learning with multiple modes is not new. In fact, in [1] (see Alg. 4, 5), the authors build on the same idea of multiple modes motivated by neural network subspaces [2]. While in this work, there is an additional term to encourage diversity, the authors in [1] report the cosine regularization in [2] does not provide substantial additional benefits. Overall, while I suspect authors may not be aware of [1], in terms of novelty, I believe the core ideas of [1] are similar to this work.



- The proposed method, MOTA, has some limitations. First, it requires knowledge of task boundaries (to discourage drift). In addition, I believe the experiments do not cover the Class-IL setup. Also, I believe several other related baselines, such as [1] and [3], should have been included in benchmarks.



- [Minor; did not play a role in my decision] I believe the paper needs additional proofreading. More specifically, there are several typos (e.g., section 1, section 2), and on page 2, the paper refers to a figure on page 8.  




**References**:  


[1] Doan, Thang et al. “Efficient Continual Learning Ensembles in Neural Network Subspaces.” ArXiv abs/2202.09826 (2022).


[2] Wortsman, Mitchell et al. “Learning Neural Network Subspaces.” ICML 2021. 


[3] Wen, Yeming et al. “BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning.” ICLR 2020.




clarity,_quality,_novelty_and_reproducibility: 

**Novelty**: As I explained before, the core ideas and main motivations of this work are already proposed. However, it may be the case that the authors were not simply aware of those works.


**Clarity & Quality**: Overall, the paper is clear. However, I believe in terms of organization and minor editorial matters, there is room for improvement. 
summary_of_the_review: Overall, I find the idea of continual learning with multiple modes interesting, I believe the paper needs improvement regarding the organization/presentation. However, I think this is a good first step.




**Update (Post-Rebuttal)**  
I would like to thank the authors for their response. Given that the authors have addressed most of my concerns and included new baselines, I would like to increase my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes the MOTA for continual learning. The idea is to train jointly a set of models for each task (like an ensemble) with an objective that encourages diversity, and then continual learning is done by coordinating the updates of the ensemble to achieve minimum model drift. Empirical analysis on task incremental learning image classification benchmarks shows MOTA achieves improvements over baselines.

Initial reviews had mixed opinions, overall they felt the perspective of balancing and coordinating networks in an ensemble for continual learning is very interesting, with one reviewer very supportive for acceptance. But others had doubts on novelty of the approach and the selection of baselines (EWC as a method published 5 years ago). The authors updated the manuscript with more baselines in one of their experiments (table 3). However, other results still use EWC as the only baseline. Furthermore, comparisons/visualisations of other ensemble-based continual learning method is missing.

For future submissions, I would suggest the authors to include more results relevant to ensemble-based continual learning in their analysis.

PS: purely looking at the score, this paper might seem to be borderline. In the end I conclude it is not borderline. See below for explanations.
justification_for_why_not_higher_score: The major concern about this paper is on comparison with the original EWC mainly, which is clearly too old as a baseline for regularisation based continual learning. Therefore it is difficult to understand how significant the gain of MOTA is.
Decision: Reject