Title: Asymptotic analysis of shallow and deep forgetting in replay with neural collapse
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: continual larning, neural collapse, deep learning

Abstract: Neural networks exhibits two forms of forgetting: deep (loss of feature separability) and shallow (classifier misalignment). We analyze the effect of replay buffers on these phenomena through an asymptotic study of feature geometry under Neural Collapse. Our results show that replay reliably mitigates deep forgetting but fails to prevent shallow forgetting, as classifier weights converge to buffer-based rather than true class statistics. We extend the NC framework to continual learning, including the multi-head setting, and characterize how buffer size, weight decay, and feature-norm growth determine feature–head alignment. The analysis further reveals that multi-head models induce structurally lower-rank feature spaces and that weight decay has divergent effects on separability across learning regimes. Finally, we establish a formal connection between continual learning and out-of-distribution detection. Empirical evaluations on standard benchmarks support the theoretical predictions and quantify the persistence of shallow forgetting across buffer sizes.

TL;DR: We analyze continual learning in the long-training limit, showing via Neural Collapse that replay preserves feature separability but causes head–feature misalignment, explaining why deep forgetting is mitigated while shallow forgetting persists.