SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison of the Multi-Event VideoQA benchmarks.
Datasets	# Avg.
Events/Video	Video
Length	#Event
Classes	#Videos	#QAs
STAR[42]	2.7	12s	157	22K	60K
AGQA v2[14]	6.8	30s	157	9.7K	2.27M
EgoTaskQA[16]	5	25s	793	2K	40K
NExT-QA[43]	8.8	44s	50	5.4K	52K
For each benchmark, we follow standard protocols outlined by prior works for data processing, metrics, and settings to ensure fair comparisons. For video feature extraction, we use frozen S3D model [46] pre-trained on HowTo100M[32] on STAR dataset and C3D[38] feature on EgoTaskQA dataset following[47, 16]. We also try Faster-RCNN model[37] pretrained on the Visual Genome[22] and CLIP(ViT-B/32) for more challenging AGQA and NExT-QA datasets. We employ a standard 2-layer, 8-head Transformer Encoder-Decoder with hidden size  $D$  of 512 as the backbone for our Glance-Focus model. According to the statistics of common event categories, the query number  $N$  is set to 10 since the maximum average number of events per video in the current dataset is approximately 8.8. And the event class number  $C$  is set to the ground truth overall class number in the dataset or 100 if it is unknown by default. For training details, we use dropout of 0.1, and initialize model weights using Xavier init[13]. Adam optimizer[20] is used with a learning rate of 5e-6 to optimize model parameters. All experiments are conducted on an NVIDIA GeForce RTX 3090Ti GPU. More details can be found in the supplementary material.
4.3 Comparison with State-of-the-arts
We conduct a comprehensive comparison with current state-of-the-arts (SOTA) VideoQA methods, including traditional spatial-temporal models and large vision language models. We compare our Glance-Focus model (shorted as GF model in the following) with them on all four datasets. The results are shown in Table 2, 3, 4 and 5 respectively. Overall, our proposed GF model achieves SOTA results on all benchmarks, and obtains significant advantages in both the STAR and the EgoTaskQA dataset. We will analyze different datasets one by one as follows. Note that in Table 2-5 the red numbers highlight the significant gains compared to prior SOTA results.
For STAR dataset, our proposed model obtains excellent reasoning performance shown in Table 2. When facing all four different types of questions, GF significantly surpasses the previous SOTAs, especially on the Sequence category. Such results reflect that our model has strong temporal contextual modeling ability. What's more, note that our model trained without ground-truth event labels, i.e., GF(uns), has achieved comparable results with the one trained under supervision, i.e., GF(sup). It indicates that our proposed unsupervised memory generation in the Glance stage can effectively extract key events with strong discriminability.
Table 2: QA accuracies of SOTA methods on STAR test set (* indicates the val set results).
Model	Interaction	Sequence	Prediction	Feasibility	Mean
ClipBERT[25]	39.81	43.59	32.34	31.42	36.70
RESERVE-B[50]	44.80	42.40	38.80	36.20	40.50
Flamingo-9B[1]	-	-	-	-	43.40
AIO[40]*	47.53	50.81	47.75	44.08	47.54
Temp[ATP][3]*	50.63	52.87	49.36	40.61	48.37
MIST[10]*	55.59	54.23	54.24	44.48	51.13
GF(uns)	51.91	63.06(+8.83)	54.89	45.57	53.86(+2.73)
GF(sup)	56.10	61.27(+7.04)	52.65	45.74	53.94(+2.81)
We also evaluate our model in the normal split of the EgoTaskQA dataset. From Table 3, we can find that our model outperforms SOTA methods on almost all question types. Our unsupervised GF model has strong performance especially on the intent and explanatory question types. These questions evaluate the understanding of the humans daily activities, which are all necessary for building the household robots. And the model trained under supervision achieves strong performance especially on object and predictive question types, which indicates that by explicitly modeling the events happening in the video, the model can obtain stronger event semantic understanding ability, namely, to recognize the interacted objects and to predict the next action.
Table 3: QA accuracies of SOTA methods on EgoTaskQA normal split.
Category	VisualBERT[26]	ClipBERT[25]	HCRN[23]	GF(uns)	GF(sup)
Scope	world	39.73	42.15	44.27	44.65	46.53
intent	44.51	40.94	49.77	52.07(+2.30)	51.88
multi-agent	26.29	27.63	31.36	33.17	34.31
Type	descriptive	41.99	38.45	43.48	44.48	46.68(+3.20)
predictive	30.37	31.50	36.56	39.00	40.36(+3.80)
counterfactual	41.99	46.75	48.00	46.66	46.80
explanatory	37.42	42.39	40.60	42.45	42.18
Semantic	action	15.02	22.91	14.92	15.68	15.68
object	23.26	21.80	45.31	45.97	50.98(+5.67)
state	59.20	54.36	68.28	70.44	71.23
change	68.27	66.58	67.38	68.69	69.71
Overall	open	24.62	27.70	30.23	31.26	32.79(+2.56)
binary	68.08	67.52	69.42	69.80	70.25
all	37.93	39.87	42.20	43.06	44.27(+2.07)
As for AGQA dataset, from Table 4, our model also achieves the SOTA result with object-level video features, i.e., GF(sup) with Faster R-CNN[37] backbone. And the one trained with S3D features also gains comparable results. We show several representative reasoning types of the dataset. Our model obtains better results on some challenging question types, such as obj-rel composition questions and temporal superlative questions like "the longest...". Overall, our model shows more clear advantage over SOTA on the challenging Open questions, compared to the relatively simple Binary questions for which the model could gain a  $50\%$  accuracy rate by blind guessing. The complete breakdown results on all kinds of question types and a more thorough analysis can be found in the Supplementary Material.
Table 4: QA accuracies of SOTA methods on AGQA v2 test set.
Method	HME[6]	HCRN[23]	AIO[40]	Temp[3]	MIST[10]	GF(uns)	GF(uns)	GF(sup)
Backbone	C3D	C3D	ViT	CLIP	CLIP	S3D	FasterRCNN	FasterRCNN
obj-rel	37.42	40.33	48.34	50.15	51.68	52.93	54.31	54.96(+3.28)
superlative	33.21	33.55	37.53	39.78	42.05	42.78	42.90	44.62(+2.57)
sequencing	49.77	49.70	49.61	48.25	67.24	53.03	51.97	53.24
exist	49.96	50.01	50.81	51.79	60.33	58.31	58.08	59.13
duration	47.03	43.84	45.36	49.59	54.62	50.86	52.02	52.80
act. recog.	5.43	5.52	18.97	18.96	19.69	22.08(+2.39)	16.38	14.17
open	31.01	36.34	-	-	50.56	53.06	55.66	56.07(+5.51)
binary	48.91	47.97	-	-	58.28	53.61	53.52	54.17
all	39.89	42.11	48.59	49.79	54.39	53.33	54.59	55.08
We also evaluate our model on the challenging NExT-QA dataset based on different video backbones. Note that the NExT-QA dataset does not provide event annotations. Therefore, we exploit the proposed unsupervised memory generation in the Glance stage. As shown in Table 5, without event-level supervision or applying any extra action detection model, the unsupervised GF models with the same backbone as other methods still achieve SOTA performances. Such results further justify the strong multi-event reasoning ability of our method, considering that NExT-QA is currently the most complex Multi-Event VideoQA benchmark with longer video duration (44s) and richer events (8.8 actions per video).
What's more, we also report the breakdown results by the different question types. According to the results, GF has strong event reasoning ability especially on Descriptive (Acc@D) questions, improving the accuracy by  $3.61\%$ , and also on event relationship reasoning, achieving  $2.31\%$  accuracy gain on Causal (Acc@C) questions. As recommended in ATP[3], we additionally report results on its proposed ATP-hard subset of the NExT-QA validation set. This subset filters out those "easy" questions that can be answered with a single frame. The results in Table 5 show that our GF models significantly surpass HGA[17] models on this subset by improving  $5.22\%$  accuracy overall. The above results indicate that compared to HGA and other methods that directly model the correspondence between question and long video through co-attention, our method using event memory as an intermediate bridge to establish such correlation for reasoning is more efficient.
Table 5: Accuracy (%) comparison on NExT-QA validation set and the ATP-hard subset[3]. Acc@C, T, D, denote accuracy for Causal, Temporal, and Descriptive questions respectively.
Methods	Backbone	val	val (ATP-hard subset)
Acc@C	Acc@T	Acc@D	Acc@All	Acc@C	Acc@T	Acc@All
ATP[3]	CLIP	48.30	49.30	65.00	49.20	19.60	22.60	20.20
VQA-T[47]	S3D	49.60	51.49	63.19	52.32	-	-	-
IGV*[28]	C3D	48.56	51.67	59.64	51.34	-	-	-
Temp[ATP][3]	CLIP	53.10	50.20	66.80	54.30	38.40	36.50	38.80
HGA[17]	C3D	46.26	50.74	59.33	49.74	43.30	45.30	44.10
MIST[10]	CLIP	54.62	56.64	66.92	57.18	-	-	-
GF(uns)	S3D	51.78	54.71	63.96	54.62(+2.30)	48.28	49.95	48.96
GF(uns)	C3D	53.59	55.71	66.80	56.33(+4.99)	48.50	49.52	48.92
GF(uns)	CLIP	56.93(+2.31)	57.07	70.53(+3.61)	58.83	48.65(+5.35)	50.27(+4.97)	49.32(+5.22)
4.4 Model Analysis
In this section, we evaluate the effectiveness of our proposed modules by comparing different variants of the GF models and various baselines. We first introduce the baselines and GF variants as follows. Baselines: 1) MDETR[18] which utilizes same 2-layer Transformer Encoder-Decoder architecture. 2) An Oracle model with ground-truth event labels concatenated to the questions as textual input as an upper-bound model. GF Variants: 1) Glance-only. Model which uses standard cross-attention to replace the multi-level attention strategy. 2) Detect-Focus. Apply a frozen action detection model, such as SlowFast[7], to replace the Glance stage of the model. We try to keep the model parameters consistent.
Effect of memory prompting. Firstly, the performance of the Oracle model significantly improves compared to the baseline model. It convinced us that event memories do play an important prompting role in the Multi-Event VideoQA task. Besides, it can be found that with event memory generation at the glancing stage, the Glance-only model outperforms the MDETR baseline and the model using a frozen action detection (Detect-Focus) with a large gap on both datasets. Such results reflect the effectiveness of our proposed memory-prompting strategy. Compared to additionally fine-tuning an action detection model, our proposed Glance strategy can adaptively generate key event/action features of the long videos without any annotations.
What's more, we evaluate the effectiveness of different loss functions that are designed during the event memory extraction in Table 7. We carefully evaluate all pairs of losses on STAR, EgoTaskQA and AGQA datasets. Firstly, the model with complete losses achieves the best results on both unsupervised and supervised settings of all datasets. It indicates that each loss plays its role in generating the event memories. Secondly, under the unsupervised setting, the  $\mathcal{L}_{iou}$  shows an
important role in controlling the temporal prediction of the events. For supervised settings, the  $\mathcal{L}_{cls}$  shows a more important role for the event semantic supervision helps cross-modal grounding. Please see more discussions on the loss functions in the Supplementary Material.
Effect of multi-level attention. From Table 6, compared to the Glance-only model using standard cross-attention, the Glance-Focus model with our designed multi-level attention shows stronger reasoning ability on both STAR and EgoTaskQA datasets. Taking an example from STAR dataset, we further visualize the attention weights in different levels in Figure 3, the darker the color, the greater the weight. The attention map in the first row shows the correspondence of the question embeddings with the generated memories. In this level, the model focuses on the most related event "Throwing a towel". The attention map in the second row is the correspondence between the focused event and the
Table 6: Module ablations on STAR and EgoTaskQA datasets.
Model	STAR	EgoTaskQA
MDETR	46.26	39.68
Detect-Focus	49.85	42.62
Glance-only	53.12	42.70
Glance-Focus	53.94	44.27
Oracle(Upper)	55.19	49.52
video embeddings. At this level, the model correctly attends to the most related keyframes for answer prediction.
Table 7: Ablation study of the loss functions on all datasets.
Uns/Sup	\( {\mathcal{L}}_{cls}^{uns} \)	\( {\mathcal{L}}_{iou} \)	\( {\mathcal{L}}_{cert} \)	\( {\mathcal{L}}_{cls}^{sup} \)	\( {\mathcal{L}}_{L1} \)	AGQA	STAR	EgoTaskQA
Uns	✓	✓	54.12	53.62	42.69
Uns	✓	✓	52.89	51.35	41.77
Uns	✓	✓	54.27	53.72	42.85
Uns	✓	✓	✓	54.59	53.86	43.06
Sup	✓	54.26	53.69	43.87
Sup	✓	53.19	53.03	42.64
Sup	✓	✓	55.08	53.94	44.27
Figure 3: Visualization of the event memory generation and the multi-level attention map.
Our proposed Glance-Focus model provides an innovative approach to the challenging Multi-Event VideoQA task. With a set of event memories as prompts, the model can efficiently establish the correspondence between high-level questions and low-level video frames. Our experiments has shown that event memories play a crucial role in locating question-related visual clues for reasoning. The proposed Glance-Focus strategy can obtain a complete and compact representation of videos without additional annotations, which can be extended to more video comprehension tasks, such as video captioning/summarization. We hope this study can inspire more future research in the field of Multi-Event Video Understanding.
This work is partially supported by National Key R&D Program of China No. 2021ZD0111901, and Natural Science Foundation of China under contracts Nos. U21B2025, U19B2036.
Conference on Computer Vision, pages 305-321, 2018.
[47] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1697, 2021.
[48] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988, 2023.
[49] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6281–6290, 2019.
[50] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Multimodal neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[51] Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhenxin Xiao, Xiaohui Yan, Jun Yu, Deng Cai, and Fei Wu. Long-form video question answering via dynamic hierarchical reinforced networks. IEEE Transactions on Image Processing, pages 5939-5952, 2019.
[52] Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8746-8755, 2020.