Title: Preserving Representation In Continual Learning via Feature-Preserving Fine-Tuning
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual learning, representation learning, fine-tuning, foundation models

Abstract: In real-world applications, deep learning models must continually adapt to sequentially arriving tasks without access to previous data. Although pre-trained foundation models show generalisation and zero-shot abilities, fine-tuning them in a continual learning setting often leads to representation degradation. In this study, we firstly systematically evaluate several recent feature-preserving fine-tuning methods (L2-SP, FTP, WiseFT and ImpReg) in continual learning scenario using a large scale pre-trained foundation model. We further explore the 
effectiveness of full fine-tuning (FullFT) versus parameter-efficient fine-tuning (PEFT) and propose a novel two-stage fine-tuning strategy, 
PEFT+Cons, designed to balance stability and plasticity by combining PEFT with task-specific knowledge consolidation. Extensive experiments on the CIFAR-100 and ImageNet-R benchmark datasets demonstrate that our proposed PEFT+Cons approach effectively prevents representation forgetting while enhancing task-specific knowledge retention.