Title: CoFiCL: Coarse-to-Fine Continual Learning with Hierarchical Knowledge
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual Learning, Vision-Language Models, Knowledge Transfer

Abstract: Vision-language models (VLMs) such as Contrastive Language-Image Pre-trained model (CLIP) show strong generalization but suffer from catastrophic forgetting in continual learning. Many existing methods only use simple category prompts like "a photo of a {class name}. This ignores the fine grained knowledge that could enrich semantic representations and support transfer. In this work, we present CoFiCL, a coarse to fine continual learning framework. The method separates coarse category knowledge and fine conceptual knowledge, which are naturally distinguished in the CLIP text space. The coarse path uses task specific adapters to align visual features with category prompts. The fine path introduces prototype based contrastive learning over language model generated descriptions, which capture semantic relations across tasks. At inference, the two paths are fused to produce the final prediction. Experiments on multiple benchmarks show that CoFiCL improves both forward and backward transfer. The results demonstrate that hierarchical knowledge can be effectively disentangled and used to enhance continual learning in vision language models.