Title: CATFormer: When Continual Learning Meets Spiking Transformers With Dynamic Thresholds
Primary area: applications to neuroscience & cognitive science
Keywords: Spiking neural networks, Lifelong learning, Transformer-based SNNs, Neuromorphic chips, Spiking threshold

Abstract: Although deep neural networks perform extremely well in controlled environments, they fail in real-world scenarios where the data isn't available all at once, and the model requires an update to adapt itself to the new data distribution, which might or might not follow the initial distribution. Previously acquired knowledge is lost during such subsequent updates from new data. a phenomenon commonly known as catastrophic forgetting. In contrast, the brain can learn without such catastrophic forgetting, irrespective of the number of tasks it encounters. Existing spiking neural networks (SNNs) for class-incremental learning (CIL) suffer a sharp performance drop as tasks accumulate. We here introduce CATFormer (Context Adaptive Threshold Transformer), a scalable framework that overcomes this limitation. We observe that the key to preventing forgetting in SNNs lies not only in synaptic plasticity, but in modulating neuronal excitability too. At the core of CATFormer is the Dynamic Threshold Leaky Integrate-and-Fire (DTLIF) neuron model, which leverages context-adaptive thresholds as the primary mechanism for knowledge retention. This is paired with a Gated Dynamic Head Selection (G-DHS) mechanism for task-agnostic inference. Extensive evaluation on both static (CIFAR-10/100/Tiny-ImageNet) and neuromorphic (CIFAR10-DVS/SHD) datasets reveals that CATFormer outperforms existing rehearsal-free CIL algorithms across various task splits, establishing it as an ideal architecture for energy-efficient and true class incremental learning.

TL;DR: A spiking transformer with dynamic thresholds for scalable class incremental learning.