Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives (2024)

Abstract: Humans use multiple senses to comprehend the environment. Vision and language are two of the most vital senses since they allow us to easily communicate our thoughts and perceive the world around us. There has been a lot of interest in creating video-language understanding systems with human-like senses since a videolanguage pair can mimic both our linguistic medium and visual environment with temporal dynamics. In this survey, we review the key tasks of these systems and highlight the associated challenges. Based on the challenges, we summarize their methods from model architecture, model training, and data perspectives. We also conduct performance comparison among the methods, and discuss promising directions for future research.

heading: 1 Introduction
text: Vision and language constitute fundamental components of our perception: vision allows us to perceive the physical world, while language enables us to describe and converse about it. However, the world is not merely a static image but exhibits dynamics in which objects move and interact across time. With the temporal dimension, videos are able to capture such temporal dynamics that characterize the physical world. Consequently, in pursuit of endowing artificial intelligence with human-like perceptual abilities, researchers have been developing Video-Language Understanding models that are capable of interpreting the spatio-temporal dynamics of videos and the semantics of language, dating back to the 1970s (Lazarus, 1973; McGurk and MacDonald, 1976). These models are distinctive from image-language understanding models, since they exhibit an additional ability to interpret the temporal dynamics (Li et al., 2020).
They have demonstrated impressive performance in various video-language understanding
∗Corresponding author
tasks. These tasks evaluate video-language models from coarse-grained to fine-grained understanding capacity. For example, for coarse-grained understanding, text-video retrieval task assesses the model’s ability to holistically associate a language query with a whole video (Han et al., 2023). For more fine-grained understanding capacity, a video captioning model is required to understand the overall and detailed video content, then describe the content in concise language (Abdar et al., 2023). Fine-grained understanding in video questioning answering remains a difficult task, where a model needs to recognize minute visual objects or actions, and infers their semantic, spatial, temporal, and causal relationships (Xiao et al., 2021).
In order to effectively perform such videolanguage understanding tasks, there are three challenges that video-language understanding works have to explore. The first challenge lies in devising an appropriate neural architecture to model the interaction between video and language modalities. The second challenge is to design an effective strategy to train video-language understanding models in order to effectively adapt to multiple target tasks and domains. The third challenge is preparing highquality video-language data that fuel the training of these models.
Although a handful of recent works have tried to review video-language understanding, they mostly focus on one challenge, for example, Transformerbased (Ruan and Jin, 2022) and LLM-augmented architecture (Tang et al., 2023b) (the 1st challenge), self-supervised learning (Schiappa et al., 2023) and pre-training (Cheng et al., 2023) (the 2nd challenge), and data augmentation (Zhou et al., 2024) (the 3rd challenge). Moreover, others also focus merely on one video-language understanding task, e.g. video question answering (Zhong et al., 2022), text-video retrieval (Zhu et al., 2023), and video captioning (Abdar et al., 2023). Such a narrow focus contradicts the growing consensus advocat-
3636
ing for the development of artificial general intelligence capable of versatile adaptation to a range of tasks and domains. Consider a human interaction scenario where an individual iteratively poses questions about a video, searches for a pertinent moment, and requests a summary. Such use case necessitates a broad capability to comprehend video and language content, without being bounded by a certain task. In addition, the development of a video-language understanding system often involves a multi-step process encompassing designing a model architecture, formulating a training method, and preparing data, rather than being a singular-step endeavor. Hence, this paper aims to present a full-fledged and meaningful survey to connect the aspects of video-language understanding. Our contributions are as follows:
• We summarize the key tasks of videolanguage understanding and discuss their common challenges: intra-modal and cross-modal interaction, cross-domain adaptation, and data preparation.
• We provide a clear taxonomy of videolanguage understanding works from three perspectives according to the three aforementioned challenges: (1) Model architecture perspective: we classify existing works into Pretransformer, Transformer-based, and LLMaugmented architectures to model videolanguage relationship. In the latter category, we discuss recent efforts that utilize the advantages of LLMs to enhance video-language understanding. (2) Model training perspective: we categorize training methods into
Pre-training and Fine-tuning to adapt videolanguage representations to target downstream task. (3) Data perspective: we summarize existing approaches that curate video-language data and annotate them to fuel the training of video-language understanding models.
• Finally, we provide our prospects and propose potential directions for future research.

heading: 2 Video-Language Tasks
text: Text-video retrieval. Text-video retrieval is the task to search for the corresponding video given a language query (text-to-video), or oppositely search for the language description given a video (video-to-text). In practical applications, returning an entire video may not be desirable. Hence, video moment retrieval (VMR) has emerged with an aim to accurately locating relevant moments within a video based on user queries. VMR examines more nuanced and fine-grained understanding to capture different concepts and events in a video in order to pinpoint specific moments rather than capturing the overall theme in standard text-video retrieval. Video captioning. Video captioning is the task to generate a concise language description for a video. A video captioning model receives as input a video and optionally a language transcript transcribed from the audio in the video. Typically, a model produces a sentence-level caption for the whole video, or might also generate a paragraph as a more detailed summary. Video question answering (videoQA). Video question answering is the task to predict the correct answer based on a question q and a video v.
Query: A man speaks to children in a classroom.
Text-video retrieval
Video captioning
Caption: add butter to the macaroni and stir
YouCook2 (Zhou et al., 2018)
MSRVTT (Xu et al., 2016)
VideoQA Question: What is someone stabbing a poster with?
Answer: swordMSVD (Chen et al., 2011)
Inference VideoQA Question: Why did the girl in blue stop and turn around at the start.
Answer: Waiting for the ladyNExt-QA (Xiao et al., 2021)
Dense video captioning
A woman wearing overalls is talking to the camera. She lays out wrapping paper, showing how to wrap a toy in it. She wraps it around the toy, then tapes it up.
0s 23s 183s ActivityNet Captions (Krishna et al., 2017)
38s 48s 60s 64s
Video moment retrieval
Query: People in scuba gear are swimming around. - QVHighlights (Lei et al., 2021)
Inferential
Fine-grained
Fine-grainedBasic level
Medium level
High level
Figure 2: Level hierarchy of video-language understanding tasks.
There are two fundamental types of VideoQA, i.e. multi-choice VideoQA and open-ended VideoQA. In multi-choice VideoQA, a model is presented with a certain number of candidate answers and it will choose the correct answer among them. Openended VideoQA can be formulated as a classification problem, a generation problem, or a regression problem. Classification-based VideoQA associates a video-question pair with an answer from a pre-defined vocabulary set. Generation-based VideoQA is not restricted to a vocabulary set, in which a model can generate a sequence of tokens that represent the answer to a question. Regressionbased VideoQA is often used for counting questions, e.g. counting the repetitions of an action or counting the number of an object in a video.
Connections among video-language understanding tasks. These tasks form the three fundamental testbeds for video-language understanding capacity (see Appendix A for their examples). In Figure 2, we provide a hierarchy that describes the levelup of their video-language understanding degree. At the basic level, text-video retrieval globally associates a whole video with a textual content. In medium level, video captioning is more difficult than retrieval tasks since it needs to selectively maps entities and events within a video to the language modality. At the highest level, videoQA
explores the relation of video and language content to produce the appropriate output. Each level of video-language understanding tasks is associated with a corresponding version that demands a more inferential or fine-grained understanding, e.g. inference videoQA (Xiao et al., 2021; Li et al., 2022a) with videoQA, dense video captioning (Zhou et al., 2018b) or video chapter generation (Yang et al., 2023b) with video captioning, and video moment retrieval (temporal grounding) with text-video retrieval. These more inferential or fine-grained tasks pose more challenges and play an increasingly significant role in current research heading towards the core of human intelligence (Fei-Fei and Krishna, 2022).

heading: 3 Challenges of Video-Language Understanding
text: The discussed video-language understanding tasks present unique challenges compared with imagelanguage understanding, since a video incorporates an additional temporal channel. We summarize their important challenges as follows: Intra-modal and cross-modal interaction. While intra-modal interaction modeling within language can be directly taken from image-language understanding, intra-modal interaction modeling within video is different since it jointly consists of spa-
tial interaction and temporal interaction. Spatial interaction delves into the relationships among pixels, patches, regions, or objects within an individual frame, whereas temporal interaction captures sequential dependencies among video frames or video segments. Longer video durations amplify the complexity of temporal modeling by necessitating the recognition of more objects and events in a higher number of video frames (Yu et al., 2020; Lin et al., 2022), and reasoning their long-term dependencies (Zhao et al., 2018). Particular video domains, such as egocentric videos, also complicate temporal interaction modeling, as objects undergo drastic appearance and disappearance dynamics over time, posing challenges in capturing their relationships (Bansal et al., 2022; Tang et al., 2023a).
Given the larger semantic gap for videolanguage compared to image-language, crossmodal interaction plays a crucial role in videolanguage understanding. The interaction between visual and language features is pivotal for aligning the semantics of video and text query to associate them for text-video retrieval, or identifying relevant parts to answer the question and writing the caption in videoQA and video captioning, respectively. In addition, incorporating the interaction of motion and language features can mitigate the extraction of noisy information from videos (Ding et al., 2022). Lin et al. (2022) also discover that the interaction between audio and language features can compactly capture information related to objects, actions, and complex events, compensating for sparsely extracted video frames. Cross-domain adaptation. Given the infinitude of online videos, that our video-language understanding model will encounter testing scenarios which are identically distributed to our training data is an impractical assumption. Moreover, with the advent of LLM-augmented models that can tackle a variety video-language understanding tasks (Li et al., 2023a,d), it is currently more advisable to train a model that can effectively adapt to multiple tasks and domains than to obtain a model which specializes in a specific understanding task. Furthermore, since a video can be considered as a sequence of images, training a model on video-text data is more computationally expensive than imagetext data. Combined with the large-scale of recent video-language understanding models (Jiang et al., 2022a; Yang et al., 2022a), there is also a need to devise an efficient fine-tuning strategy to save the computational cost of fine-tuning these models. Data preparation. Although Lei et al. (2021c) only use image-text data to train models for videolanguage understanding tasks, in essence, videotext data are crucial for the effectiveness of these models. In particular, compared with a static image, a video offers richer information with diverse spatial semantics with consistent temporal dynamics (Zhuang et al., 2023). As such, Cheng et al. (2023) find that training on videos outperforms training on images, but jointly training on both data achieves the best performance. As additional evidence, Yuan et al. (2023) shows that video-pretrained models outperform image-pretrained models in classifying motion-rich videos. However, video-text data takes up more storage cost than image-text data since a video comprises multiple images as video frames. Moreover, annotating a video is also more time-consuming and labor-intensive than annotating an image (Xing et al., 2023). Therefore, videolanguage understanding models have been limited by the small size of clean paired video-text corpora in contrast to billion-scale image-text datasets (Zhao et al., 2023). Various efforts (Zhao et al., 2023; Xing et al., 2023) have been put into devising efficient and economical methods to curate and label video-text data. Addressing challenges. These identified challenges encompass three critical perspectives: model architecture, model training, and data preparation in the field of video-language understanding.
In general, there should be a synergistic relationship among these components. Specifically, model architecture should be designed to effectively capture video-language interactions. Concurrently, model training should be tailored to enable the architecture to adapt to target domains with their captured video-language interactions. Lastly, data preparation plays a pivotal role in shaping model training, which in turn significantly impacts the development of an efficacious model architecture.

heading: 4 Model Architecture for VideoLanguage Understanding
text: Addressing the challenge of intra-modal and crossmodal interaction is the key aim in designing video-language understanding model architectures, which can be divided into Pre-transformer and Transformer-based architectures. The advent of LLMs with remarkable zero-shot capability in addressing multiple tasks led to the design of LLM-augmented architectures that exhibit cross-domain adaptation ability to various videolanguage understanding tasks.

heading: 4.1 Pre-Transformer Architecture
text: Pre-transformer architectures typically comprise unimodal video and language encoders for implementing intra-modal interactions and cross-modal encoders for cross-modal interactions. Unimodal encoders. A video encoder often encodes raw videos by extracting frame appearance and clip motion features as spatial and temporal representations, respectively. As each video frame can be considered as a single image, various works have utilized CNNs to extract spatial representations (Simonyan and Zisserman, 2014; Feichtenhofer et al., 2016; Zhao et al., 2017b). For temporal representations, the sequential nature of RNN makes it a popular choice in pre-transformer architectures (Yang
et al., 2017; Zhao et al., 2017a; Venugopalan et al., 2015; Wang et al., 2019a). Furthermore, 3D CNNs with an additional temporal channel inserted to 2D CNN have also demonstrated effectiveness in extracting spatio-temporal representations (Tran et al., 2017; Carreira and Zisserman, 2017). In addition to CNN and RNN, Chen et al. (2018), Gay et al. (2019), and Wei et al. (2017) also build graphs to incorporate intra-modal relationships among video entities such as video segments or visual objects. These graph-structured works emphasize the reasoning ability of the model architecture.
A common framework of language encoder is to extract pre-trained word embeddings such as word2vec (Kaufman et al., 2016; Yu et al., 2017) or GloVe (Torabi et al., 2016; Kiros et al., 2014), then proceed with RNN-based modules such as LSTM or GRU. Such framework is taken from language model architectures before the era of Transformer. Cross-modal encoders. Gao et al. (2017) and Zeng et al. (2017) apply element-wise multiplication to fuse the global video and question representations for video question answering. It demonstrates the advantage of a simple operation for video-language fusion. Attention has also been used to model video-language relations, in order to identify salient parts in video and language sentence (Yuan et al., 2019), or to refine the representation of the video based on the language question (Xu et al., 2017). Pre-transformer video-language works have also combined attention with a wide variety of techniques, including hierarchical learning (Baraldi et al., 2017), memory networks (Fan et al., 2019), and graph networks (Xiao et al., 2022a; Wei et al., 2023).

heading: 4.2 Transformer-based Architecture
text: Developed based on the self-attention mechanism, which exhaustively correlates every pair of input tokens with each other, Transformer-based architecture has the capacity to capture longterm dependencies and learn from web-scale data. It has demonstrated remarkable performance in many video-language tasks. Similar to the pretransformer architecture, the Transformer-based framework also comprises unimodal encoders and cross-modal encoders to model intra-modal and cross-modal interactions, respectively. For unimodal encoders, several works find vision transformer for video encoding and BERT encoder for language encoding outperform RNN- and CNN-
based encoding (Fu et al., 2021; Bain et al., 2021; Seo et al., 2022). We then summarize fundamental types of Transformer-based architectures and illustrate them in Figure 4. Shared Transformer. Motivated by the success of Transformer in language modeling (Devlin et al., 2018), Akbari et al. (2021) and Wang et al. (2023a) construct a shared Transformer encoder for videolanguage understanding. Their encoder architectures receive the concatenation of visual patches and language tokens, then jointly calculate their interactions in a BERT-based manner. Akbari et al. (2021) additionally incorporate modality embeddings which comprise three values to denote three kinds of input modalities, i.e. (video, audio, text). Stacked Transformer. Li et al. (2020) reveals that a shared Transformer encoder is weak in modeling temporal relations between videos and texts. To address this problem, they introduce a stacked Transformer architecture, with a hierarchical stack consisting of unimodal encoders to encode video and language inputs separately, and then a crossmodal Transformer to compute video-language interactions. A multitude of video-language understanding works follow such design to stack a crossmodal Transformer-based encoder above unimodal encoders (Fu et al., 2023; Li et al., 2023b; Lei et al., 2021c; Wei et al., 2022; Luo et al., 2022; Nie et al., 2022; Wei et al., 2024). To perform video captioning, Seo et al. (2022) and Luo et al. (2020) further insert a causal Transformer-based decoder that generates language tokens based on the encoded cross-modal representations. Dual Transformer. Dual Transformer architectures have been favored for text-video retrieval (Luo et al., 2022; Bain et al., 2021, 2022; Lin et al., 2022; Xue et al., 2022b). These architectures use two Transformer encoders to encode video and language separately, yielding global representations for each input modality, then applying simple operations such as cosine similarity to compute cross-modal interaction. Such a separate encoding scheme enables them to mitigate the computational cost of computing pairwise interactions between every pair of video and language inputs. They have accomplished not only efficiency but also effectiveness in text-video retrieval problems.

heading: 4.3 LLM-Augmented Architecture
text: Large language models (LLMs) have achieved impressive results in simultaneously tackling mul-
tiple NLP tasks. Recent efforts have sought to apply LLMs for video-language understanding to extend its cross-domain adaptation ability to videolanguage settings (Chen et al., 2023; Li et al., 2023a). These efforts can be categorized into two approaches. The first approach employs LLM as a controller and video-language understanding models as helping tools. The controller will call the specific tool according to the language input instruction. The second approach utilizes LLM as the output generator and seeks to align video pretrained models to the LLM. For video-language understanding, since the second approach dominates the first one with a long list of recent works (Chen et al., 2023; Li et al., 2023a; Chen et al., 2023; Li et al., 2023d; Zhang et al., 2023b; Maaz et al., 2023), we review them as follows: LLM as Output Generator. The framework comprises a visual encoder, a semantic translator, and an LLM as the output generator. Regarding visual encoder, LLM-augmented architectures often use vision transformer and CNN models of the pre-Transformer and Transformer-based architectures (Chen et al., 2023). Since an LLM has never seen a video during its training, a semantic translator is needed to translate the visual semantics of a video to the LLM’s semantics. For the translator, Video-LLaMA (Zhang et al., 2023b) and VideoChat (Li et al., 2023a) implement a QFormer as a Transformer-based module that uses a sequence of query embeddings that interact with visual features of the video to extract informative video information. Instead of Q-Former, VideoLLM (Chen et al., 2023), Video-ChatGPT (Maaz et al., 2023), and LLaMA-Vid (Li et al., 2023d) find that a simple linear projection that projects visual features into the LLM’s input dimension can achieve effective performance. Subsequently, these visual-based query embeddings or projected visual features are combined with the language instruction to become the input fed to the LLM to produce the final output.

heading: 4.4 Architecture Analysis
text: In Figure 3, we show the timeline of videolanguage understanding methodologies, categorized according to our defined architecture taxonomy and their affiliated downstream tasks. The evolution of pre-transformer models aligns with our hierarchy of video-language understanding levels, i.e. models for video captioning generally appear sub-
sequent to those for text-video retrieval, followed by the development of videoQA models. Owing to their impressive capacity, Transformer-based models capable of addressing multiple tasks have been introduced concurrently with task-specific Transformer frameworks. Recently, large language models (LLMs) have gained prominence for their superior in-context learning ability, enabling them to handle diverse tasks without fine-tuning. Consequently, new LLM-augmented architectures have emerged to utilize this capability to address multiple understanding tasks.
Among Transformer-based architectures, the dual Transformer stands out as the most effective for text-video retrieval, adeptly associating global semantics of video and language modality. On the other hand, the stacked Transformer architecture excels at facilitating intra-modal and cross-modal interactions through its specialized unimodal and cross-modal encoders. These encoders are particularly efficient at correlating video content with the question in videoQA. Additionally, for video captioning, cross-modal encoder plays a crucial role in translating video content into textual descriptions. Recent LLM-augmented models have begun to outperform Transformer-based architectures in videoQA, signalling their potential as the next frontier in video-language understanding research. We provide full details of performance in text-video retrieval, video captioning, and videoQA tasks in Table 1, 2, and 3, respectively.

heading: 5 Model Training for Video-Language Understanding
text: Model training seeks to address the cross-domain adaptation ability of video-language understanding models. To achieve this goal, pre-training strategies have been devised to gain world knowledge that generalizes across multiple scenarios, then taskspecific fine-tuning is conducted to specifically improve downstream task performance.

heading: 5.1 Pre-training for Video-Language Understanding
text: In this section, we mainly summarize pre-training strategies for video-language understanding models into three groups: Language-based pre-training. The most popular language-based pre-training task is masked language modeling (MLM) (Lei et al., 2021c; Sun et al., 2019; Cheng et al., 2023), which randomly
masks a portion of words in the language input and trains the model to predict the masked words based on unmasked language words and video entities. Instead of masking a portion of words, UniVL (Luo et al., 2020) and VICTOR (Lei et al., 2021a) discover that masking the whole language modality benefits video captioning task. MLM can be combined with other language-based pre-training task, e.g. masked sentence order modeling which is to classify the original order of the shuffled language sentences (Lei et al., 2021a).
Video-based pre-training. Video-based pretraining tasks help video-language models capture contextual information in the video modality. As a counterpart of MLM, masked video modeling (MVM) trains the model to predict the portion of masked video entities based upon the unmasked entities and language words. The continuous nature of videos leads to different choices of video entities, such as frame patches (Li et al., 2020) or video frames (Fu et al., 2021). In terms of the training objective, Li et al. (2020) use L2 regression loss to train the model to predict pre-trained features of the masked video frames extracted by ResNet and SlowFast models, while Fu et al. (2021) use cross-entropy loss to train the model to predict the masked visual tokens, which are quantized by a variational autoencoder from visual frame patches.
Video-text pre-training. Video-text pre-training is crucial for a model to capture video-language relation. Xue et al. (2022b), Gao et al. (2021), and Bain et al. (2021) utilize a framework of video-text contrastive learning to produce close representations for semantically similar video and language inputs. These works focus on creating a joint semantic space that aligns separate representations of video and language. Instead of separate representations, Tang et al. (2021), Fu et al. (2021), and Li et al. (2023b) enable video and textual representations to interact with each other and use a single token to represent the cross-modal input, which is forwarded to predict whether the video-text pair is matched or not. In these two pre-training frameworks, not only video-text data but also image-text data are utilized during pre-training, in which an image is considered as a video with a single frame.
Contrastive learning has revealed promising results (Lin et al., 2022; Gao et al., 2021; Xue et al., 2022b; Nguyen et al., 2022; Nguyen and Luu, 2021; Nguyen et al., 2024c,a, 2023a). MLM has contributed to enhancing VideoQA since the task re-
sembles MLM in predicting the language word given a video-language pair (the question is the language input in videoQA). Compared to these pre-training strategies, MVM does provide performance gain for video-language understanding but its gain is less significant. For more details about pre-training, please refer to (Cheng et al., 2023).

heading: 5.2 Fine-tuning for Video-Language Understanding
text: Task-specific fine-tuning is commonly used by preTransformer architectures to train from scratch since these models do not have sufficient parameter capacity to learn generalizable features through pretraining. It is also widely adopted by Transformerbased architectures to improve the performance for a specific downstream task. Moreover, LLMaugmented architectures also utilize instruction tuning as a variant of fine-tuning, to adapt from the visual and audio spaces to the LLM language space. Fine-tuning strategies. Normally, all of the model parameters are updated during fine-tuning (Gao et al., 2017; Xu et al., 2019; Anne Hendricks et al., 2017; Nguyen et al., 2023b). However, in cases computational resources or training data are limited, only adaptation layers such as lowrank adapters (Pan et al., 2022; Yang et al., 2022a; Nguyen et al., 2024b) or learnable prompt vectors (Ju et al., 2022) are fine-tuned to reduce training cost or prevent overfitting. Such risks also apply for LLM-augmented architectures discussed in Section 4.3, since LLMs exhibit a billion scale of parameters, thus incurring excessively huge cost if full finetuning is conducted. For such models, Zhang et al. (2023b) and Li et al. (2023d) design a two-stage instruction tuning strategy which only fine-tunes the semantic translator. The first stage trains the model to generate the textual description based on the combined video and the language instruction, in order to align visual representations extracted by the visual encoder with the language space of LLM. The second stage is often performed on small-scale video-text pairs manually collected by the authors to further tailor the output features of the translator towards the target domains.

heading: 6 Data Perspective for Video-Language Understanding
text: In this section, we analyze data preparation approaches for video-language understanding models, and provide details of the datasets in Appendix B.

heading: 6.1 Data curation
text: Manual collection. To curate video-language data, multiple works search for publicly available online videos, which exhibit a wide diversity of content. Video-language datasets with online videos are mostly aimed for pre-training models to learn generalizable knowledge, e.g. HowTo100M (Miech et al., 2019) and YT-Temporal-180M (Zellers et al., 2021), or they can also be used for fine-tuning, e.g. MSRVTT (Xu et al., 2016) and YouCook2 (Zhou et al., 2018a). To satisfy a certain requirement, videos different from the online ones can be inherited from existing datasets, e.g Xiao et al. (2021) utilize 6,000 videos from VidOR dataset and (Li et al., 2022a) inherit 546,882 videos from Kinetics700 since they describe scenes of daily life and real world, respectively. Apart from making use of existing datasets’ and online videos, videos can also be recorded by human annotators to enable quality control (Goyal et al., 2017; Damen et al., 2022). Data augmentation. Rather than manually collecting videos from external sources, Xing et al. (2023) and Jiang et al. (2022c) explore data augmentation techniques which are particularly designed for videos. In detail, their TubeTokenMix mixes two videos in which the mixing coefficient is defined upon the temporal dimension, and their temporal shift randomly shifts video frame features backward or forward over the temporal dimension. These techniques outperform standard augmentation approaches for image data, such as CutMix (Yun et al., 2019), Mixup (Zhang et al., 2017), and PixMix (Hendrycks et al., 2022).

heading: 6.2 Label annotation
text: Manual annotation. Several works (Li et al., 2022a; Lei et al., 2021b; Xiao et al., 2021) use human annotators since they provide high-quality labels. However, such approach is expensive, particularly when dealing with video data. For example, annotating QVHighlights dataset (Lei et al., 2021b) costs approximately $16,000 for 10K videos and 3 months to complete. Similarly, NExT-QA (Xiao et al., 2021) needs 100 undergraduate students and 1 year to annotate only 5K videos. Automatic generation. Directly taking language transcripts of YouTube videos as textual labels could reduce annotation cost (Miech et al., 2019; Xue et al., 2022a; Zellers et al., 2021). However, these labels have been shown to be grammatically incorrect and temporally misalign with the video
content (Tang et al., 2021). Motivated by the success of LLMs, Zhao et al. (2023) train a system consisting of a TimeSformer-L visual encoder and a GPT-2XL decoder to write dense captions for videos. Moreover, Li et al. (2023a) use GPT-4 to generate summaries for movie synopses.

heading: 7 Future Directions
text: Fine-grained understanding. Existing methods excel at video-language understanding at a coarsegrained level, enabling effective responses to questions like “what is” or the recognition of global events without significant difficulty (Xiao et al., 2021). Nevertheless, limiting comprehension to this coarse level could hinder practical utility of existing systems. In real-world scenarios, a user might require a precise timestamp and location of an object within a video (Jiang et al., 2022b), or request the AI agent to forecast potential alternative events, which is a common need in predictive analytics (Xiao et al., 2021; Li et al., 2022a). These tasks necessitate an advanced understanding and inference capability regarding the causal and temporal relationships present in a video. At present, models exhibit a constrained visio-linguistic capacity to engage in temporal reasoning, categorizing them as image-sequence-and-language models rather than video-language models (Kesen et al., 2023). Therefore, future research in this direction deserves more attention and exploration. Long-form video-language understanding. Current understanding systems have demonstrated remarkable performance on short video clips lasting several seconds. However, they tend to struggle when switching to long-form videos which last several minutes or hours. To enhance the applicability of these systems, it is essential to enhance their capability of understanding long-form videos. Current approaches mainly feature reducing computational cost through architectures more efficient than Transformer-based ones such as state space models (Yang et al., 2024; Li et al., 2024), which can be considered as linear RNN with specifically designed fixed weights, or compensating sparsely extracted video frames with additional information (Lin et al., 2022). In general, how to effectively model long-form videos and adapt them to the joint context with language deserves more attention. Trustworthiness of video-language understanding models. Although modern video-language understanding systems have demonstrated remarkable
performance, their black-box nature undermines our trust to deploy them. In particular, we still do not precisely understand what part of the video a videoQA model looks at to answer the question (Li et al., 2022b), or how video and language semantic information flows into the common representation space of the video retrieval model (Jia et al., 2022). Furthermore, adversarial noise sensitivity or hallucination of video-language understanding models are also open problems. Future trustworthiness benchmarks such as (Xiao et al., 2023a; Wang et al., 2021a) for video-language understanding are of great significance towards practical systems.

heading: 8 Conclusion
text: In this paper, we survey the broad research field of video-language understanding. Particularly, we categorize related video-language understanding tasks and discuss meaningful insights from model architecture, model training, and data perspectives. We thoroughly analyze each perspective, and finally conclude with promising future directions. We hope our survey can foster more research towards constructing effective AI systems that can comprehensively understand dynamic visual world and meaningfully interact with humans.

heading: 9 Limitations
text: Although we have sought to comprehensively analyze the literature of video-language understanding, we might not fully cover all of the tasks, model architectures, model training, and data perspectives. Therefore, we complement the survey with a repository https://github.com/nguyentthong/ video-language-understanding. The repository comprises the latest papers, datasets, and their open-source implementations. We will periodically update the repository to trace the progress of the latest research.

heading: 10 Acknowledgement
text: This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-TC2022-005).

heading: A Examples of Video-Language Understanding tasks
text: In this appendix, we provide examples of video-language understanding tasks in Figure 5 and 6.

heading: B Analysis of Video-Language Understanding datasets
text: Due to page limit, details of the datasets for video-language understanding tasks are listed in Table 4. We categorize all the datasets according to their tasks that they support. While datasets for downstream tasks and fine-tuning have been consistently developed, those for pre-training emerge subsequently to the introduction of the Transformer architecture. Although pre-training and downstream video-language understanding datasets pursue different goals, they predominantly originate from the Internet. Regarding downstream datasets, more recent ones aim to present new technical challenges, such as evaluating reasoning and inference abilities (Xiao et al., 2021; Li et al., 2022a), or examining long-form modeling capacity of video-language understanding models (Mangalam et al., 2023).

heading: Future Work
text: We thoroughly analyze each perspective, and finally conclude with promising future directions. We hope our survey can foster more research towards constructing effective AI systems that can comprehensively understand dynamic visual world and meaningfully interact with humans