Title: TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video summarization, Multimodal

Abstract: The exponential growth of video content highlights the importance of video summarization, a task that efficiently extracts key information from long videos. However, existing video summarization studies face inherent limitations in understanding complex, multimodal videos.  This limitation arises primarily because these studies often focus on unimodal or bimodal analysis, failing to capture the intricate interplay between visual cues, speech transcripts, and audio events. To overcome these limitations, we propose a novel architecture, TripleSumm, which adaptively weights and fuses the contributions of the three modalities at the frame level. Furthermore, a significant bottleneck for research into multimodal video summarization has been the lack of comprehensive benchmarks. Addressing this bottleneck, we introduce MoSu (Most Replayed Multimodal Video Summarization), the first large-scale benchmark that provides all three modalities. Our proposed TripleSumm demonstrates its superiority by achieving state-of-the-art performance by a large margin on four video summarization benchmarks, including MoSu.

TL;DR: We propose TripleSumm, a frame-level adaptive multimodal fusion model, and introduce MoSu, the first large-scale benchmark with all three modalities, achieving state-of-the-art video summarization performance.