Title: Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual Learning, CLIP, Low-Rank Adaptation

Abstract: Continual learning (CL) aims to accumulate knowledge from sequential tasks without catastrophic forgetting. Vision–language models like CLIP, with strong generalization, are widely used for CL. Existing methods often adapt isolated PTM components, adding inference complexity and limiting PTM improvement, or rely on replay, stored information, or assumptions, incurring high costs and limited applicability. To advance models as continual learners, we explore CL via natural, efficient PTM updates instead of complex task-specific additions. We thus study continual low-rank learning and systematically analyze how LoRA ranks and placements affect *learning* and *forgetting*. We find that a relatively *higher-rank* LoRA improves task learning (*i.e.*, *plasticity*) but increases forgetting, while a relatively *lower-rank* LoRA reduces forgetting (*i.e.*, *stability*) but limits adaptation. Crucially, we find a *plasticity–stability balance* tied to rank across parameters and tasks, with *moderately small ranks* maximizing CL benefits. Motivated by this, we propose **Co**ntinual **Dy**namic **R**ank-Selective LoR**A** (**CoDyRA**), which continually updates PTMs with LoRA adapters of adaptively optimized rank. While the new-task objective drives learning, CoDyRA adaptively minimizes ranks with *sparsity-promoting regularization* to reduce interference and forgetting, achieving a plasticity–stability balance tailored to different parameters and tasks. Adaptively selected and minimized LoRA ranks keep the updated model closer to its previous state while learning new tasks. CoDyRA enables efficient CL as a sequence of LoRA-based tasks without storing past data, task information, or relying on assumptions. It preserves the original model architecture and deployment pipeline, adding no inference overhead. Extensive experiments show CoDyRA improves new representations while retaining old knowledge, achieving state-of-the-art results.