Title: SPKLIP: Aligning Spike Video Streams with Natural Language
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Neuromorphic Computing, Multimodal Learning, Energy-efficient AI, spiking neural network

Abstract: Spike cameras offer unique sensing capabilities but their sparse, asynchronous output challenges semantic understanding, especially for Spike Video-Language Alignment (Spike-VLA) where models like CLIP underperform due to modality mismatch. We introduce SPKLIP, the first architecture specifically for Spike-VLA. SPKLIP employs a hierarchical spike feature extractor that adaptively models multi-scale temporal dynamics in event streams, and uses spike-text contrastive learning to directly align spike video with language, enabling effective few-shot learning. A full-spiking visual encoder variant, integrating SNN components into our pipeline, demonstrates enhanced energy efficiency. Experiments show state-of-the-art performance on benchmark spike datasets and strong few-shot generalization on a newly contributed real-world dataset. SPKLIP's energy efficiency highlights its potential for neuromorphic deployment, advancing event-based multimodal research. The source code and dataset are available at [link removed for anonymity].

TL;DR: This paper introduces SPKLIP, the first end-to-end architecture for aligning sparse, asynchronous spike video streams with natural language, demonstrating SOTA performance and energy-efficient potential for neuromorphic vision.