Text-Conditioned Resampler For Long Form Video Understanding
Authors: name: Bruno Korbar; name: Yongqin Xian; name: Alessio Tonioni; name: Andrew Zisserman; name: Federico Tombari
Published: 2023-12-19 06:42:47

Abstract: Videos are highly redundant data source and it is often enough to identify a
few key moments to solve any given task. In this paper, we present a
text-conditioned video resampler (TCR) module that uses a pre-trained and
frozen visual encoder and large language model (LLM) to process long video
sequences for a task. TCR localises relevant visual features from the video
given a text condition and provides them to a LLM to generate a text response.
Due to its lightweight design and use of cross-attention, TCR can process more
than 100 frames at a time allowing the model to use much longer chunks of video
than earlier works. We make the following contributions: (i) we design a
transformer-based sampling architecture that can process long videos
conditioned on a task, together with a training method that enables it to
bridge pre-trained visual and language models; (ii) we empirically validate its
efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art
on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks
which require longer video contexts and that can thus be used effectively for
further evaluation of long-range video models.