Video-Panda: Parameter-efficient Alignment for Encoder-free
  Video-Language Models
Authors: name: Jinhui Yi; name: Syed Talal Wasim; name: Yanan Luo; name: Muzammal Naseer; name: Juergen Gall
Published: 2024-12-24 18:59:56

Abstract: We present an efficient encoder-free approach for video-language
understanding that achieves competitive performance while significantly
reducing computational overhead. Current video-language models typically rely
on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B
parameters), creating a substantial computational burden when processing
multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment
Block (STAB) that directly processes video inputs without requiring pre-trained
encoders while using only 45M parameters for visual processing - at least a
6.5times reduction compared to traditional approaches. The STAB architecture
combines Local Spatio-Temporal Encoding for fine-grained feature extraction,
efficient spatial downsampling through learned attention and separate
mechanisms for modeling frame-level and video-level relationships. Our model
achieves comparable or superior performance to encoder-based approaches for
open-ended video question answering on standard benchmarks. The fine-grained
video question-answering evaluation demonstrates our model's effectiveness,
outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key
aspects like correctness and temporal understanding. Extensive ablation studies
validate our architectural choices and demonstrate the effectiveness of our
spatio-temporal modeling approach while achieving 3-4times faster processing
speeds than previous methods. Code is available at
https://github.com/jh-yi/Video-Panda.