Same Accuracy, Twice As Fast: Continual Learning Surpasses Retraining From Scratch

Authors: Eli Verwimp, Guy Hacohen, Tinne Tuytelaars
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning

Abstract:
Continual learning aims to enable models to adapt to new datasets without losing performance on previously learned data, often assuming prior data is no longer available. However, in many practical scenarios, both old and new data are accessible. In such cases, good performance on both datasets is typically achieved by abandoning the model trained on the previous data and re-training a new model from scratch on both datasets. This training from scratch is computationally expensive. In contrast, methods that leverage the previously trained model are worthy of investigation as they could significantly reduce computational costs. Our evaluation framework quantifies the computational savings of such methods while maintaining or exceeding the performance of training from scratch. We identify key optimization aspects - initialization, regularization, data selection, and hyper-parameters - that can each contribute to reducing computational costs. For each aspect, we propose effective first-step methods that already yield substantial computational savings. By combining these strategies, we achieve up to 2.7x reductions in computation time across various computer vision tasks, highlighting the potential for further advancements in this area.

Decision: reject
Reviewer scores: 3, 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729075078195

thread_type:
official_review

<thread_object>
creator: reviewer_wjks

<content>
summary:
Overall, this paper first defines a new setting of "continual learning", where the old data, the current data, and the model that has been trained on the old data are all accessiable. After that, it presents a bag of tricks that can enable the model to achieve much faster convergence in this new setting of "continual learning".

strengths:
Overall, I believe that the problem that this paper aims to handle is an important problem. In other words, when we already have a model that has been trained over part of the dataset, it is a waste to simply re-train everything from scratch.

Also, the paper present this problem in a well-motivated way.

weaknesses:
(See questions)

questions:
Overall, I believe that this paper is not ready to be accepted. Below are my concerns.

1. The main concern I have w.r.t. this paper is over its defination that its setting is a ``continual learning'' setting. Specifically, correct me if I am wrong, yet, based on my understanding of continual learning, the catastrophic forgetting is always a key problem in this area, no matter over which of its sub-settings such as class-incremental learning, domain-incremental learning, and so on. Yet, with the access of the old data, the paper seems to define a continual learning setting that is essentially different from the general continual learning concept. This worries me that whether this paper fits itself in a wrong area for the research community. For example, if my understanding is not wrong, warm starting is a method that is kind of just solving the problem that this paper aims to solve. Yet, no one would recognize warm starting as a continual learning method.

2. Furthermore, the paper highlights that it makes several "first-step methods". Yet, I am confused over the first step here. If I am not wrong, the paper seems to be highly inspired by previous methods and thus seems to be closely related to them. For example, the initialization part seems to be closely related to Ash & Adams (2020), the batch composition part seems to be closely related to Hacohen & Tuytelaars (2024), and so on. Specifically, while I can admit that adapt some method to a new area is a contribution, as mentioned in my previous concern, I doubt whether the authors do propose a "new" setting, not to say that the contribution behind adaptation can be small.

3. My last concern lies in the generalizability of the proposed method. Specifically, it seems to me that the method just try some approaches and observe that they work on a few (small) datasets. I am not sure whether this can well justify and demonstrate the effectiveness of the proposed method.

Given my above concerns, I believe that this paper is not ready to be accepted in its current version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729828560867

thread_type:
official_review

<thread_object>
creator: reviewer_2cbz

<content>
summary:
This paper explores the problem of efficient model training through the lens of continuous training. It focuses on four aspects: initialization, regularization, data selection and hyper-parameter tuning. For each aspect, it introduces specific techniques to enhance the training efficiency. The experiments are performed on several image classification datasets using a ResNet-18 network. By integrating all the proposed methods, the paper demonstrates improved performance while reducing training costs.

strengths:
1. It is interesting to explore efficient training from the perspective of continuous training. 
2. This paper provides an initial investigation using different training techniques and shows their effectivenss. The strategies employed perform well even in multi-step training scenarios.
3. This paper is easy to follow.

weaknesses:
1. This paper only demonstrates its feasibility on small-scale models and datasets. However, efficiency gains are more critical when dealing with large models and datasets. Please also provide the validation on larger model and datasets. It would be better to see results on ImageNet-1k with ViTs and comparing with training from scratch or other pre-train based continual leanring baselines such as [A]. 
2. The use of the term "continual learning" may be misleading, as it conficts with its wildly accepted meaning in the field. It would be better to use other phrase like "continuous training" instead. Additionally, providing a clear definition of the task is necessary to avoid confusion.
3. There are some discrepancies in the reported results. For instance, in Table 2(b), the cost reduction ratios are ×1.61 and ×1.45 for CIFAR-100 (70+30) with full data, and ×4.74 and ×4.34 with 25\% data. However, in Table 1, the reported values are ×1.96, ×1.69, and ×5.73, ×5.32, respectively. These inconsistencies should be addressed and clarified.

[A] SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model. Zhang et al. ICCV 2023.

questions:
Line 315, "More formally, they define learning speed ls of a sample $x_j$ as the epoch $e$ in which sample is classified correctly" is unclear. $e$ is not used in Equation 6.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474685669

thread_type:
official_review

<thread_object>
creator: reviewer_5btg

<content>
summary:
This paper tackles the practical aspect of continual learning setup, wherein access to both new and old data exists and the major cost of training newer models is on the computation side. With the set of previously known techniques, authors show that when used,  in combination they provide significant speed-up in the training of a new model under the described continual learning paradigm. The experiments are done on the standard datasets which shows the validity of the claims.

strengths:
1. This method shows the combination of model initialization, learning rate scheduling, batch composition and the loss function, affect the rate at which the model learns on new data.
2. Though the method is simple and a combination of known techniques, it provides good improvement on some of the datasets.
3. The paper is well-written and easy to read.

weaknesses:
1. The major issue with this work is how generalizable are the claims on the varied set of backbone architecture and datasets. For eg. ViT backbones, downstream tasks, like object detection or segmentation, or varied datasets like ImageNet or wider domain shifts like in WiLDS[1]. It will make the claims stronger with more extensive results. As section 4.3 suggests the claim might not hold under domain shifts.
2. Will the method benefit already existing CL methods(ones mentioned in Sec1.2)? if the claims are general, it should benefit the CL methods and one can hope for the best of both, performance-boost and computation efficiency.
3. The comparisons with existing CL methods are missing in the experiments in terms of performance and compute efficiency. $L_r$ can be computed for them too. This makes the assessment of the contribution difficult.
4. The test accuracy is reported for combined new and old data. It will be better to see the accuracy on them separately.


[1]:https://arxiv.org/pdf/2211.14238

questions:
See the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642244742

thread_type:
official_review

<thread_object>
creator: reviewer_vbzf

<content>
summary:
This paper considers a new continual learning setting where the old data and new data are both accessible for training and the efficiency is the main concern. For each task, the straightforward baselines include training the model from scratch with both old and new data and fine-tuning the model trained on old data with new data. The former serves as a strong baseline considering the accuracy but is computationally expensive. The latter suffers from a much worse accuracy. Thus, this paper investigates a bag of tricks used in SGD training to improve both the accuracy and efficiency of the way that fine-tunes the model pretrained on old data. Empirical study verifies the effectiveness of proposed method.

strengths:
- The paper is generally well-written and easy to follow. 

- The empirical results are good. 

- It verifies a bag of tricks which may help the practitioners to continually train a model with satisfactory accuracy and efficiency.

weaknesses:
- Some details are not clear. For example, in Fig. 3, the curve corresponding to the legend "L2" means "scratch + L2" or "naive + L2"? If it is the later, how about the curve of "scratch + L2"? 

  I don't quite understand the descriptions from Line 415-419. I am confused about how the authors compute different costs. Besides, for 
  Fig. 6, I don't understand what the curve between $T_{i-1}$ and $T_i$ means. And I am not clear how the authors draw the curve. 

- There is a typo in Eq. (6), the $x$ and $y$ on the right side of equal sign should be $x_j$ and $y_j$. 

- For the batch composition trick, it can be shared across the fine-tuning way and the scratch way. From the empirical results shown in Table 1, applying this trick to the "continuous" and the "scratch" yields the same speed considering $L_{100}$ and almost the same speed considering $L_{99}$. Also, by comparing "init+reg" and "init+reg+data" (continuous), it seems that the batch composition trick does not work. To me, the results are reasonable. I don't understand why batch composition can yield larger gain when combined with continuous model compared to that combined with scratch model. Do the authors have any explanations for this?

questions:
- The authors applied a bag of tricks which may be proposed in previous works to the considered continual learning setting. Can the authors provide any insights why those tricks work in the continual learning setting? Are there any other tricks which may also benefit the accuracy and efficiency of continual learning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688571444

thread_type:
official_review

<thread_object>
creator: reviewer_wpkw

<content>
summary:
The work "proposes" a continual learning framework/analysis based on optimization to reduce the computational cost during the continual learning of models.

strengths:
The authors conduct a thorough analysis of the SGD optimization scheme on a continual learning (CL) setup and open new insights into a new direction. The analysis is limited and not thorough. The literature on CL has moved way ahead, considering the core data and model assumptions that this work has considered. However, such an analysis is also important.

weaknesses:
The core contributions of this work are completely minimal and rely on the analysis of standard deep-learning optimization schemes, especially with regard to the parameter update rule, in a continual learning (CL) scheme. The results are not dense and the experimental setup is very basic. Rigorous experiments are to be done on a wide scale. The authors claim that "old" and "new" data are usually available, which is very impractical by today's standards. Please take a look at recent works [1, 2, 3]. However, many works have proposed CL methods with no access to past data + (data and parametric)-efficient ways. This work doesn't meet the standards of an ICLR submission, however. The analysis is decent, but the experiments are not thorough with no clear "proposed method". My suggestion would be to conduct such analysis on a larger scale, covering a wide range of cases - training setups (CIL, TIL), model backbones, challenging scenarios, etc.



[1] Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: CVPR. pp. 139–149

[2] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free
continual learning. In CVPR, pages 11909–11919, 2023.

[3] Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-free continual learning. In ICCV, pages 11847–11857, 2023.

questions:
1. It is not clear to me how important saving "computational time" is. Could the authors benchmark against FLOPs? 
2. If the "old" data isn't available, can the authors benchmark their work against prompt-based CL methods like L2P [1]? Or can a similar analysis be done with a ViT backbone?
3. Why do the authors use a 70+30 split on CIFAR-100? How about a uniform split in tasks?
4. How would the analysis hold with rapid domain shifts? 
5. Is there a typo in Fig 6 caption/analysis? In the main text (4.2), a CIFAR-100 (70+30) was used.
6. How is the analysis with an Adam/AdamW optimizer? Is this analysis model invariant and holds everywhere?



[1] Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: CVPR. pp. 139–149
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new setup for continual learning where both old and new data, as well as a pre-trained model, are accessible. It demonstrates that combining existing techniques significantly accelerates training in this setup, validated through experiments on standard datasets. The reviewers highlights several major concerns about the practicality and generalisability of the proposed approach. Firstly, the assumption that both old and new data are typically available is considered unrealistic in many real-world scenarios, where retaining access to old data is often restricted due to privacy, storage, or regulatory constraints. Secondly, doubts are raised about the generalisability of the claims, as the experiments lack sufficient evidence to demonstrate the effectiveness of the proposed methods across a diverse range of backbone architectures and datasets. Lastly, the use of the term "continual learning" is flagged as potentially misleading, as it diverges from the widely accepted definition in the field, which typically refers to scenarios without access to old data.