Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides an empirical study on pretraining for lifelong learning. The paper suggests that the generic pre-training implicitly alleviates the effects of catastrophic forgetting in lifelong learning and the reason is the pre-trained weights can ease forgetting by leading to wider minima. The model is evaluated on a range of datasets in CV and NLP to support the findings. 




main_review: Pros
- The paper is very well-written and easy to follow.  
The paper shows that pre-weights can alleviate catastrophic forgetting and suggests an explanation of it - pertained weights lead to wider minima. With that, the authors show that explicitly seeking flat basins during sequential fine-tuning results in even less forgetting. Again, the coherent structure and detailed analysis are a plus. 

Concerns
-  Missing baselines.  Various methods have been proposed in the continual/lifelong learning line of work. Although it is hard to exhaustively evaluate all possible methods, some popular approaches are still missing, such as iCaRL and PackNet (parameter isolation-based approaches). Therefore, the authors may group their baselines based on method types, similar to the structures in the survey paper (e.g., https://arxiv.org/pdf/1909.08383.pdf). 
- Limited setting. The paper focuses only on the Tabula rasa case, where incremental learners trained from scratch and do not require pretraining on large labeled datasets. A more realistic setting is to learn from pre-trained models. For example, a model is trained on the first 50 classes of CIFAR-100 and then incrementally learns the remaining classes. Is the paper trying to suggest CL benchmarks should use pre-trained weights instead of training from scratch? 
- The CV datasets are relatively small.  The methods are evaluated on CIFAR datasets while in the literature (again checking the survey paper above),  larger datasets like ImageNet, iNaturalist, are used for evaluation.  Can the authors provide the analysis there? 
- Other evaluation metrics, like Backward/Forward transfer? 

summary_of_the_review: The paper provides an empirical study to understand the role of pre-training in lifelong learning.  Given that the paper draws conclusion based on empirical results, it's important to have a systematic study over the existing continual learning settings and baselines (e.g.,  various training/evaluation settings,  evaluation metrics such as Backward/Forward transfer,  datasets, etc.). The current manuscript pushes one step further towards understanding the role of pretraining, but still more studies can be added.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a comparative study on the impact of using pre-trained models on task incremental learning problems. They split computer vision and NLP datasets into multiple tasks and evaluate performance between pre-trained and randomly initialised models. They conclude that task incremental learning benefits from pre-training due to the flatter loss landscapes, and propose to use an optimisation constraint to learn better models. 
main_review: Empirical analysis papers like this one, aiming to identify and understand best training practices, are sorely needed in machine learning. Authors have made substantial efforts in building multiple datasets, comparing different settings, and trying to understand what their findings mean. The proposed loss constraint solution appears to work well, and improves overall model performance. 

Despite the large amount of work carried out, I have several concerns about the proposed work and its conclusions.   
Firstly, the main conclusions of the different experiments are not providing significant novel insights. It is well known that model pre-training provides significant performance boosts and robust transfer training solutions. It is expected that pre-training would yield superior performance in datasets somewhat similar to the pre-training dataset. Furthermore, comparing model weights and loss landscape between a robust pre-trained model and a randomly initialised, potentially not converged model does not seem particularly fair. The large gap between learning accuracies in certain settings highlights this issue. The flat loss landscape is a more interesting, yet not novel, conclusion as acknowledged in section 6. 

One key limitation is that, besides analysing forgetting, the experiments proposed here provide little insights with regards, specifically, to the task incremental setting. For example, task dissimilarity and ordering could have an impact on how beneficial pre-training is. What happens if the first task is highly different from the pre-training dataset?  What is  the impact of the choice of the pre-training dataset? This was briefly discussed when comparing different BERT models, but the main conclusion was that larger and more diverse pre-training datasets are better, which is also a well known result. Authors had a great opportunity to analyse the impact of the dissimilarity between the pre-training and task specific datasets on forgetting with their 5(+)-datasets settings. It would have been very interesting to study the impact of ordering tasks differently depending on their distance to the pre-training dataset. 

Furthermore, reported results only provide overall accuracy and forgetting. Task specific results would provide much more insights, especially when dealing with heterogeneous tasks. Are observed differences consistent across tasks? Are there tasks where pre-training is detrimental, or less beneficial?  

The presentation of the paper could be improved greatly. There is a lot of content, which makes the paper extremely condensed with a lot of essential information relegated to supplementary material such as the discussion on related work. The issue is also reflected in the presentation of the results. Different experiments are reported in the same table, which is referenced multiple times at different points in the paper. This leads the reader to wonder where to look in the table, and what the SAM method is, not knowing that it will be introduced at the end of the paper. This creates confusion. 

In addition, authors provide very little information on datasets. It is not stated in the main paper why these specific datasets are chosen, whether their order was curated, and how tasks were constructed. Again, a lot of the essential information is relegated to the supplementary material when this constitutes a key contribution. 

Similarly, while there exist a large set of methods aiming to address the task incremental learning problem, authors selected 4 without clear justification. Why were these methods selected over others?

Minor comments:

How were semantically similar classes between ImageNet and CIFAR determined?

Please double check citation formats. Some were not accurately formatted in the text. 

Please add line number to facilitate the review process and point out typos.

summary_of_the_review: In summary, authors provide a study with a lot of potential, but unfortunately the paper in its current state does not provide substantially novel conclusions or innovations. While the proposed experiment are substantial and do provide additional confirmations, main conclusions are mostly well known (pre-training achieves better performance, larger datasets and models are better), or already discussed in the lifelong learning community (flat loss landscape).  The proposed SAM optimisation constraint is a direct application of another work, and its interaction with different continual learning methods, which often rely on optimisation constraints, is not discussed. 
Finally, the paper is presenting a lot of content which is severely hurting presentation and readability, with essential elements relegated to supplementary material or missing. I would recommend refocusing the work or submitting in settings where page limit is not a concern.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores the catastrophic forgetting in lifelong learning from the perspective of model initialization. It further shows that large pre-trained models have an implicit bias towards wider loss basin when fine-tuning on sequential tasks which ultimately results in less forgetting of tasks when training on subsequent tasks. Furthermore, by explicitly seeking wider loss basins during lifelong learning training as an additional optimization objective further reduces the forgetting regardless of the lifelong learning algorithm and base model used.
main_review: Strong points:
 - This paper takes a different and interesting approach to tackle one of the most important issues of lifelong learning i.e., catastrophic forgetting. As pre-training of large models on large diverse datasets has become a norm, this approach looks more practical to me than fine-grained changes in rehearsal-based and regularization based approaches.
 - The main hypothesis that pre-training helps in reducing catastrophic forgetting over a sequence of datasets/tasks has been supported theoretically and empirically with diverse quantitative results and qualitative analyses.
 - To strongly demonstrate the validity of main hypothesis, this paper provides comprehensive experiments, including both quantitative results and qualitative analysis: (1) using prominent approaches of lifelong learning in recent years consisting of rehearsal based (A-GEM, ER) and regularization based (EWC) approaches, (2) for Computer Vision (CV) and Natural Language Processing (NLP) applications, (3) on homogeneous (Split CIFAR-100, Split Cifar-50, Split YahooQA) and diverse (5-dataset, 5-dataset-NLP, 15-dataset-NLP) sequential datasets.
 - For example, it has been demonstrated that a simple fine-tuning (without regularization or rehearsal) of a pre-trained model on a sequence of diverse tasks (5-dataset-NLP) undergoes significantly less forgetting than rehearsal-based ER method with randomly initialized weights, which begs the question that if instead of specifically focusing on forgetting aspect of lifelong learning would it be more useful to learn generic features on a larger scale?
 - In case of diverse sequential datasets (5-dataset, 5-dataset-NLP) simple fine-tuning of a pre-trained model performs worse than ER method which shows that pre-trained models are susceptible to forgetting when learning on diverse tasks. However, larger models and models pre-trained with diverse pre-trained corpora undergoes less forgetting which shows that diversity of pre-trained corpus is a relevant factor during lifelong learning.
 - Experiments have been performed with multiple different sequences of tasks in each dataset to test robustness of the proposed approach.
 - Loss contour plots and Sharpness metric evaluations further validates the hypothesis that pre-trained models have bias for wider loss basins for fine-tuning tasks.
 - It has been shown that in addition to pre-trained weight initialization, it further helps in reducing forgetting if sharpness of the loss basin is taken into account as an additional training objective via Sharpness-Aware Minimization (SAM) training procedure. Experimental results confirm that SAM eases catastrophic forgetting and improves the performance of underlying model whether it is "fine-tuning of a pre-trained model" or "episodic replay (ER)".

Weak points:
 - Following the setup of experiments of ER method with and without SAM training procedure, it would be interesting to see the experiments with model initialized with pre-trained weights in ER method to see if fine-tuning of pre-trained weights and rehearsal of examples from previous tasks complement each other or not.
 - To see the effects of SAM optimization procedure qualitatively, I would recommend to plot loss contours for Task 1 (as shown in Figure 2) for Finetune + SAM models.
 - Also, it would be interesting to see loss contours for Task 1 using models trained with ER method with randomly initialized weights to compare Finetune and ER methods qualitatively.
summary_of_the_review: Although there is no novel algorithm or model introduced in this paper, but the direction it explores to minimize catastrophic forgetting in lifelong learning methods via pre-trained weight initialization is interesting and practical as large pre-trained models have become a norm because of the wide range of diverse knowledge they can capture. Even though the Finetune method is not the best performing method in all of the experiments (however it outperforms in most of the experiments), the key conclusion I can take form this paper is that instead of completely focusing on improving upon fine-grained details of forgetting aspect of lifelong learning, the focus should also be on learning more general representations as they appear to result in robust lifelong learning.
review_title: null

Meta-review: The paper presents an empirical study on the impact of pertained model on lifelong learning. It concludes that the generic pertaining can benefit the lifelong learning duet the flatter loss landscape and evaluates on CV and NLP tasks. The paper is well written with detailed analysis. However, there is concerns on its limited setting and the conclusion is known in the community and based on empirical studies.
Decision: Reject