Transformer-based Video Question Answering
Year: 2023

Abstract:
Conventional Transformer-based Video Question Answering (VideoQA) approaches generally encode frames independently through one or more image encoders followed by interaction between frames and question. However, such schema incur significant memory use and inevitably slow down the training and inference speed. In this work, we present a highly efficient approach for VideoQA based on existing vision-language pre-trained models where we concatenate video frames to a n × n matrix and then convert it to one image. By doing so, we reduce the use of the image encoder from n to 1 while maintaining the temporal structure of the original video. Experimental results on MSRVTT and TrafficQA show that our proposed approach achieves state-of-theart performance with nearly 4× faster speed and only 30% memory use. We show that by integrating our approach into VideoQA systems we can achieve comparable, even superior, performance with a significant speed up for training and inference. We believe the proposed approach can facilitate VideoQA-related research by reducing the computational requirements for those who have limited access to budgets and resources. Our code is publicly available at https://github.com/lyuchenyang/ Efficient-VideoQA for research use.

text: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 183–189 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Transformer-based Video Question Answering (VideoQA) (Xu et al., 2016; Yu et al., 2018; Xu et al., 2021b; Bain et al., 2021; Lei et al., 2022) approaches relying on large scale vision transformers (Dosovitskiy et al., 2020) have achieved strong performance in recent years. However, such approaches typically encode multiple video frames separately through one or more Image Encoders (Lei et al., 2021; Luo et al., 2021; Xu et al., 2021a; Arnab et al., 2021;
∗*corresponding author
Zhong et al., 2022) followed by interaction with question representations. This requires significant memory use and inevitably slows down training and inference speed. In order to reduce the computational cost required for modeling video representations from frames, we propose to arrange the frames sampled from one video as a single image. Specifically, we sample n2 frames from one video and concatenate them as a single image with n× n grids.
Figure 1 shows the difference between our method (right) and conventional methods (left). In general, conventional approaches either independently encode video frames (Luo et al., 2021; Xu et al., 2021a; Lei et al., 2021; Bain et al., 2022) which require N forward passes, or encode the sequence of all patches in video frames (Bain et al., 2021; Arnab et al., 2021) which quadratically increases the computational cost in the attention modelling. Both types of aforementioned encoding approaches can be expected to negatively impact training and inference speed, whereas our proposed method reduces this need substantially, now requiring only a single forward pass.
Our method diverges from previous approaches in two ways: 1) it fully relies on existing available pre-trained vision-language models such as CLIP (Radford et al., 2021) without need for extra
183
pre-training (Bain et al., 2021; Lei et al., 2022); 2) it considers a multi-frame video as a single image, dispensing with the need for positional embedding at the frame level (Bain et al., 2021), so only minor modifications to pre-trained models are necessary.
More importantly, our approach has three advantages: 1) higher computational efficiency1; 2) less memory use - our approach only uses an Image Encoder a single time; 3) our approach can be easily scaled up for large numbers of frames for long videos. Our approach also models a multipleframe video as a single image while still (partially) maintaining the temporal structure of the original video.
To validate the effectiveness of our approach, we conduct experiments on two benchmark VideoQA datasets: MSRVTT-MC (Xu et al., 2016; Yu et al., 2018) and TrafficQA (Xu et al., 2021b). Results show that our approach achieves comparable or even superior performance compared to existing models with nearly 4× faster training and inference speed and vast reduction in memory use (30%). Our contribution can be summarised as follows:
• We propose a novel approach combining video frames as a single image to accelerate VideoQA systems ;
• Experimental results on MSRVTT-MC and TrafficQA show that our proposed approach achieves competitive performance with faster training-inference speed and lower memory use;
• We include additional experiments investigating options for arrangement of video frames for VideoQA;

heading: 2 Model Architecture
text: In this section, we introduce details of our approach, of which an overview is shown in Figure 2.

heading: 2.1 Vision Transformer
text: Generally Vision Transformer (ViT) (Dosovitskiy et al., 2020) flattens a single image to m nonoverlapping patches v = {p0, p1, ......, pm−1}. All patches are fed into a linear projection and then regarded as discrete tokens in (Vaswani et al., 2017; Radford et al., 2018; Devlin et al., 2019) followed
1For example, the computational cost of Bain et al. (2021); Arnab et al. (2021) scales up quadratically w.r.t. the number of image patches whereas ours is invariant w.r.t. the number of image patches.
by transformer-based modeling (Vaswani et al., 2017; Dosovitskiy et al., 2020). The output feature is vh = {h0, h1, ......, hm−1}, where Eh ∈ Rm×d, d is the dimension of the output feature for each patch.
For encoding a multiple-frame video, suppose that we have an input video V = {v0, v1, ......, vn−1} with n frames, so for each frame vi ViT flattens it to m non-overlapping patches vi = {pi,0, pi,1, ......, pi,m−1}. The patches for each frame are concatenated to form a sequence of patches:
V = {p0,0, ......, p0,m−1, ......, pi,0, pi,1, ......, pi,m−1, ......} (1)
which are fed into (a) linear projection(s) followed by transformer-based modeling (Vaswani et al., 2017; Dosovitskiy et al., 2020)2. We thus obtain frame-level representations:
V h = {v0, v1, ......, vn−1} (2) where V h ∈ Rn×d. 3

heading: 2.2 Interaction with Question Representations
text: For a natural language question Q = {w0, w1, ......, wk−1} consisting of k words, we use a textual transformer to encode Q to obtain a sentence-level representation Qh ∈ R1×d. Since in this work, we mainly focus on reducing the computational cost of encoding videos, we perform simple interactions between video representations V h and question representations Qh:
2Frames can be encoded separately through one or more Image Encoder (Luo et al., 2021; Xu et al., 2021a; Bain et al., 2022) or all patches can also be concatenated and passed into one Image Encoder (Bain et al., 2021; Arnab et al., 2021)
3Patch-level representations V h = {h0,0, ......, h0,m−1, ......, hi,0, hi,1, ......, hi,m−1, ......} are used in (Lei et al., 2021; Bain et al., 2021)
V h ′ = MULTI-HEAD-ATTENTION(Qh, V h, V h)
(3) where V h ′ ∈ R1×d is the question weighted representations and MULTI-HEADATTENTION (Vaswani et al., 2017) performs attention between V h (key and value) and Qh (query).

heading: 2.3 Frames Transformation
text: When encoding multiple frames, the encoding schema in 2.1 incurs significant memory use and additionally impedes training and inference speed (Lei et al., 2021; Luo et al., 2021; Xu et al., 2021a; Lei et al., 2022). Therefore, we propose a novel strategy to reduce the computational cost associated with encoding videos by combining all frames into a single image arranged by n×n grids. Practically, we arrange all frames to a matrix, M , in which each entry corresponds to a frame. For a video with n × n frames, we put each frame into Mi,j in a specific order. For example, frames can be arranged in M via ascending or descending order (either vertically or horizontally) based on its index in the video.4 Next, we convert M to a single image. Therefore, regardless of how many frames we use, the number of tokenized patches (image tokens) is always a constant number, resulting in a computationally more efficient VideoQA system.

heading: 3 Experiments

heading: 3.1 Datasets
text: We conduct experiments on two benchmark datasets for VideoQA: MSRVTT-MC (Xu et al., 2016; Yu et al., 2018) and TrafficQA (Xu et al., 2021b), which are multi-choice VideoQA datasets – each video in MSRVTT-MC is associated with 5 candidate options whereas TrafficQA provides 4 options for each question. We follow the standard data split for MSRVTT-MC (Xu et al., 2016; Yu et al., 2018), where evaluation data have 2,990 videos. TrafficQA contains 62,535 QA pairs and 10,080 videos. We follow the standard split of TrafficQA: 56,460 QA pairs for training and 6,075 QA pairs for evaluation.
4The effect of various arrangement orders is shown in Sec 3.5.

heading: 3.2 Experimental Setup
text: We use CLIP ViT-B/16 (Radford et al., 2021) 5 to initialize our IMAGE-ENCODER and TEXTENCODER. We evenly sample 9 frames from the videos in MSRVTT-MC and TrafficQA for the main experiment. We train our model for 20 epochs with a learning rate of 1e-6. The training batch size is 16. We use a maximum gradient norm of 1. The optimizer we used is AdamW (Loshchilov and Hutter, 2019), for which the ϵ is set to 1× 10−8.

heading: 3.3 Evaluation Results
text: We show the evaluation results on MSRVTTMC (Xu et al., 2016; Yu et al., 2018) in Table 1. Furthermore, we conduct experiments on TrafficQA (Xu et al., 2021b) and the results are shown in Table 2. We present the results of separately encoding video frames (MULTI-FRAME) as in Figure 1 (left) and our approach that combines multiple video frames into a single image (SINGLEFRAME). For SINGLE-FRAME, the frames are arranged in a matrix via horizontally descending order. The evaluation results show that our approach SINGLE-FRAME achieves comparable and even improved performance relative to strong baselines including VideoCLIP (Xu et al., 2021a), All-inOne (Wang et al., 2022), Singularity (Lei et al., 2022) and CMCIR (Liu et al., 2022). SINGLEFRAME obtains a significant speed up (approaching ×4) compared to MULTI-FRAME approach while maintaining competitive performance. The memory use of SINGLE-FRAME is only 30% of MULTI-
5https://openai.com/blog/clip/
FRAME, which are compared on Nvidia GTX 3090. The results on two benchmark datasets have shown the effectiveness of our approach for improving the computational efficiency while maintaining accuracy of VideoQA systems.

heading: 3.4 Effect of Number of Frames
text: We investigate the effect of the number of video frames used by our approach during the training and inference process. The results are shown in Figure 3. We compare the performance of MULTIFRAME and SINGLE-FRAME for number of frames ranging from 1 to 256 in Figure 3. Results show that: 1) Both MULTI-FRAME and SINGLE-FRAME systems can benefit from more video frames; 2) SINGLE-FRAME is capable of achieving comparable and even better performance against MULTIFRAME; 3) MULTI-FRAME costs much more computational time than SINGLE-FRAME especially when using a large number of video frames. Therefore, our proposed SINGLE-FRAME approach is able to achieve higher efficiency as well as competitive accuracy.

heading: 3.5 Effect of Frame Order
text: We investigate the effect of the arrangement of video frames used to form a single frame. The
6For SINGLE-FRAME with a number of frames that is not a square number we up-sample it to the closest square number. For example, a SINGLE-FRAME that deals with 2 frames with index of {0, 1}, we upsample it to {0, 0, 1, 1} as 2×2 images.
results of both the 9 frame and 16 frame configurations are shown in Table 3. We report the results of VERTICAL, HORIZONTAL and MATRIX via either ASCENDING or DESCENDING order.7 The results in Table 3 reveal that both VERTICAL and HORIZONTAL perform worst, and this is likely due to configurations distorting the visual information since they essentially squeeze the video frames either vertically or horizontally. The MATRIX arrangement performs substantially better especially MATRIX (HORIZONTAL-DESCENT) and HORIZONTAL generally yielding better performance compared to VERTICAL under the MATRIX arrangement.

heading: 4 Conclusion and Future Work
text: In this paper, we propose a highly efficient method for VideoQA where we combine multiple video frames into one single image. By adapting our approach, the computational cost of VideoQA systems can be significantly reduced. To validate the effectiveness of our approach, we conduct experiments on two benchmark datasets, MSRVTT-MC and TrafficQA. Results show that our approach
7Examples are shown in Appendix A.1.
achieves competitive performance and faster training and inference speed (nearly 4× faster) and less memory consumption (30%). Our approach provides a way of significantly accelerating training and inference. In the future, we aim to explore how to adapt our approach to VideoQA with longer videos and additional video-related NLP tasks.

heading: Acknowledgements
text: This work was funded by Science Foundation Ireland through the SFI Centre for Research Training in Machine Learning (18/CRT/6183). We thank the reviewers for helpful feedback.
Limitations
The two VideoQA datasets used in experiments are associated with relatively short videos. Therefore it would be better if more experiments could be conducted on VideoQA datasets with long videos to verify the effectiveness of our approach on a wider range of VideoQA tasks. Although the proposed approach in this paper can also be used in other video-language tasks, our experiments focuses on a specific video-language task - VideoQA. Experiments on more video-language tasks are needed to show that our approach are also effective in other video-language tasks.

heading: A Appendix
text: A.1 Examples of frames with different arrangement order
We present some examples of frames with different arrangement order in Figure 4, where we use 9 frames as examples. The arrangement orders are: (1) HORIZONTAL. (2) VERTICAL. (3) MATRIX (HORIZONTAL-ASCENT). (4) MATRIX (HORIZONTAL-DESCENT). (5) MATRIX (VERTICAL-ASCENT). (6) MATRIX (VERTICALDESCENT). The corresponding video frame indices are shown in Figure 5.

heading: Future Work
text: In the future, we aim to explore how to adapt our approach to VideoQA with longer videos and additional video-related NLP tasks