VideoLights: Feature Refinement and Cross-Task Alignment Transformer for
  Joint Video Highlight Detection and Moment Retrieval
Authors: name: Dhiman Paul; name: Md Rizwan Parvez; name: Nabeel Mohammed; name: Shafin Rahman
Published: 2024-12-02 14:45:53

Abstract: Video Highlight Detection and Moment Retrieval (HD/MR) are essential in video
analysis. Recent joint prediction transformer models often overlook their
cross-task dynamics and video-text alignment and refinement. Moreover, most
models typically use limited, uni-directional attention mechanisms, resulting
in weakly integrated representations and suboptimal performance in capturing
the interdependence between video and text modalities. Although large-language
and vision-language models (LLM/LVLMs) have gained prominence across various
domains, their application in this field remains relatively underexplored. Here
we propose VideoLights, a novel HD/MR framework addressing these limitations
through (i) Convolutional Projection and Feature Refinement modules with an
alignment loss for better video-text feature alignment, (ii) Bi-Directional
Cross-Modal Fusion network for strongly coupled query-aware clip
representations, and (iii) Uni-directional joint-task feedback mechanism
enhancing both tasks through correlation. In addition, (iv) we introduce hard
positive/negative losses for adaptive error penalization and improved learning,
and (v) leverage LVLMs like BLIP-2 for enhanced multimodal feature integration
and intelligent pretraining using synthetic data generated from LVLMs.
Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks
demonstrate state-of-the-art performance. Codes and models are available at
https://github.com/dpaul06/VideoLights .