Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: Incremental training via fine-tuning of large-scale deep learning models often face challenging issues like catastrophic forgetting. As a remedy, this paper proposes three methods 1) Fisher Shut-off, 2) Fractional Data Retention and 3) Border Control for commercial deep learning fine-tuning scenarios. The effectiveness of the proposed remedies are tested on a toy dataset. 
strengths_and_weaknesses: Strength:

Incremental learning on commercial applications is an important topic. 

Weakness:

The originality of this paper is not sufficient. While certain ideas are definitely interesting (like the Fisher shut off), most of the proposed strategies already exist, so the value of information from this paper would be quite limited. 

Experiment section must be further enriched. I don't quite get the logic of illustrating the effectiveness of the proposed methods on a toy dataset, while one of the selling points of this paper is "commercial fine tuning applications".

I think the presentation of this paper can be much improved. Details should be added to the included strategies, e.g., section 3.3
questions: N/A.
limitations: N/A.
review_title: null

final_score: rating: 1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations
review_content: summary: This paper introduces three simplified methods for incremental learning. The three methods are Fisher Shut-off, Fractional Data Retention and Border Control, which are all adapted from previous works.  Fisher Shut-off select weights in models that take part in fine-tuning. Fractional Data Retention and Border Control retains a fraction of data from earlier tasks and dataset. All the methods are illustrated with a toy dataset, and the author claims that they can be used in real world datasets.
strengths_and_weaknesses: Strengths: 

The proposed methods are simple and easy to implement.

Weakness:

1. Lack of originality, the methods proposed in this paper are mainly built on other works. The differences are not clearly shown in the paper.

2. The evaluation is on a toy dataset and it is too simple to support the claims that the proposed methods can be used in real-world applications. 

3. The data selection method designed in Fractional data retention and Border control is not well studied and evaluated in this paper. Deep learning data selection is studied in many topics, i.e, active learning, model repair, and test data selections. To evaluate the effectiveness of the Fractional data retention and Border control, the authors should show that their methods are better than random selection and other commonly used data selection methods. [[1](https://arxiv.org/pdf/1708.00489.pdf),), [2](https://arxiv.org/pdf/1906.03671.pdf)]

4. The presentation of this paper is poor.
questions: 1. All three methods are adapted from previous works, what are the key differences?

2. Experiments are conducted on a toy dataset. Please provide experimental results on real-world datasets to support the claims in this work.

3. Even for the toy dataset experiment, 
    - Why there are no existing techniques used for comparison? 
    - I am not sure if the experiments in Table 1, i.e, FDR 10% and FDR 20%, or BC (topk=5) and BC (topk=10), have a different percentage of retained data. If so, a direct comparison between them is not fair.
    - Experiments with incremental learning without FS are essential.

3. FDR requires data clustering, which technique do you use in the experiment? Do different clustering techniques largely affect the result?

4. In BC, can you provide some proof that selecting top-k farthest data points are more effective than random selection? Since the paper is claimed for real-world datasets with low-quality data, it is likely that the top-k farthest data points are out of distribution, which provides limited help.
limitations: The limitation of this paper is partly addressed in Conclusion.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This paper aims to address the incremental learning of pre-trained models for practical commercial scenarios. No novel method is proposed, and this work seems like a course report. 
strengths_and_weaknesses: Pros:
- None

Cons:
- There is not even a novel method proposed to address the problem of focus.
- Plenty of existing works in the NLP and even the CV domain have been ignored. See the references [1-6] below.
- All the results are evaluated on very small synthetic datasets; none of the existing benchmarks has been investigated. Moreover, the existing state-of-the-art baselines have not been compared against.

[1] Jang, Joel, et al. "Towards Continual Knowledge Learning of Language Models." International Conference on Learning Representations. 2022. 
[2] Lin, Bill Yuchen, et al. "On Continual Model Refinement in Out-of-Distribution Data Streams." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.
[3] Ermis, Beyza, et al. "Continual Learning With Transformers for Image Classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[4] Jin, Xisen, et al. "Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora." Challenges & Perspectives in Creating Large Language Models (2022): 
[5] LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
[6] Pelosin, Francesco, et al. "Towards exemplar-free continual learning in vision transformers: an account of attention, functional and weight regularization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: None
limitations: See the Cons above.
review_title: null

Meta-review: metareview: This paper motivates problems related to fine tuning of pre-trained deep learning models for commercial applications and proposes three solutions for incremental learning: Fisher Shut-off, Fractional Data Retention and Border Control).  The reviewers thought the work was well-motivated and they were in agreement that this is a timely and important topic.   However, they all found the novelty to be too low and the experiments unconvincing for NeurIPS.  Therefore the recommendation is to reject the paper.  
confidence: Certain
Decision: Reject