Glance and Focus: Memory Prompting for Multi-Event Video Question Answering
Authors: Ziyi Bai
Ruiping Wang
Xilin CHEN
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Question Answering; Multi-Event Reasoning; Spatial-Temporal Reasoning

Abstract: Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents’ ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance- Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross- attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.

Glance and Focus: Memory Prompting for Multi-Event Video Question Answering
Ziyi Bai Ruiping Wang 1.2, Xilin Chen 1.2
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS),
Institute of Computing Technology. CAS, Beijing. 100190, China
2 University of Chinese Academy of Sciences, Beijing, 100049, China ziyi. @ vipl.ict.ac.cn, /wangruiping. xlchen/ fcchen/@ict.ac.on
Abstract
Video Question Answering (VideoQA) has emerged as vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using series of episode memories as anchors to quickly locate question-relant for reasoning, To mimic this effective reasoning strategy, propose the Glance- Focus model. One simple way is to apply an action detection model to predict set of actions as key memories. However, these actions within closed set vocabulary are hard to generalize to various video domains. Instead of that, we Decoder to generate set of dynamic event memories the glancing stage. Apart from using supervised bipartite matching to obtain the further design unsupervised memory generation method
key moments
train Encoder-Decoder
event memories, to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act bridge establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross- attention mechanism. We conduct extensive experiments four Multi-Even VideoQA benchmarks including STAR, EgoTaaskQ, AGQA, and NExT-Qn QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https:///githbb wewpttuu.ccoo/yy// www.co/te///////cc Betr200/Glance-oooo
Introduction
For intelligent agent that assist humans in completing daily-life tasks, complex event reasoning ability particularly important. Multiple Event Video Question Answering (Multi-Event VideoQA) task [14] 42 16 43] has recently been proposed. These benchmarks require agents to reason about complex videos depicting multiple human-object interactions (i.e., events or situations). Furthermore, wide scope of event-centric questions are 114) ranging from event semantic descriptions to event sequencing and dependencies to comprehensively evaluate an agent's ability to understand human behavior.
Recently, the emerged large vision- lision-language models 13 8 25 47 50] have demonstrated impressive large-scale cross-modal
generalization capabilities in VideoQA tasks. By leveraging pre- pre-training pairs, these models can establish correlations between questions and videos to find answers effectively. Such approaches proves effective for short videos containing single or few events with one-step
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

02
Pz-4
Py--
Py--2
Py--1
Fyy
Pr-4
Vr-d
Dy-2
Phy-1
Episocle Memoory
Q: What did the person do the blanket before taking the clothes?
Glance: Generating Episode Memory
X2
X2
Polting the binglen
before
Py-3
Sefore
video
12
before
Py-T
person
toklnge
question
person
taking question
dinhes
the
A: Took. X
cloches
A: Put Down.
before
Q: What ... bafora laking the clathes?
A: Took.
taking oothes?
A: Put Down.
Focus: Reasoning with Memory Prompt
(a) VideoQA through cross-modality
correspandence modeling
(b) VideoQA through Glance-Focus
Figure 1: Comparison of different VideoQA solutions. The cross-modal common feature space is visualized. (a) Cross-modal correspondence modeling. Weak correspondence between the question with high-level event concepts and low-level lengthy video leads to wrong answer. (b) Human- like Glance-Focus Strategy. Glancing stage: set of episode memories are generated. Focusing stage: memory prompts are used as anchors to quickly locate the keyframes for reasoning.
questions like "What is someone doing?". However, Multi-Event VideoQA poses distinct challenge. As shown in Figure Ta), due to the longer video duration and more events involved, it is challenging for the model effectively find the visual clues from the lengthy video content. Additionally, the event-centric questions often include high-level event concepts while the semantics of video frames relatively low-level, making even more difficult to directly locate key moments related the question.
This is very similar to the difficulties humans encounter when addressing reading comprehension tasks, where we must quickly extract question-related context from long paragraphs. To ensure fast and effective reasoning, it is common for humans employ strategy of first glancing through the story, summarizing the storyline in our mind, and then using this episode memory to quickly focus on the relevant context when answering questions.
We mimic such a reasoning strategy to tackle the challenging Multi-Event VidcoQA task with our proposed Clance-Foocu model. It comprises two stages: glancing and focusing. The purpose of the glancing stage is to generate a set of episode memories that can efficiently summarize the multiple events occurring in the video. A naive way is to directly train action detection model to obtain set of actions. However, this would result unnecessary computation, and most importantly, it may not be practical for encapsulating all possible actions in the real world using closed vocabularies.
Instead, we adopt a more flexible approach that automatically obtains of dynamic event memories using a Transformer Encoder- Decoder architecture (39). For datasets with closed event vocabularies, we employ bipartite matching between the extracted memories and ground truth labels to explicitly obtain event memories. Apart from supervised event matching approach, additionally design an unsupervised event generation way to get rid of the event-level annotations of the video. We impose individual certainty and global diversity constraints during event generation ensure they can effectively summarize the video content.
Finally, since these generated memories are still independent events, we reorganize them into complete event memory prompts by arranging them in sequence order with temporal information remaining. As shown in Figure [b)) since these memory prompts have high-level event semantics, they play an anchor role the multi-modal common feature space to help quickly locate the related video content according to the question. Such the example in Figure [bb, we design multi-level cross

---

attention mechanism to first focus on the event "putting down the blanket" before "taking the clothes" according to the question, then focus on the related video content for answer prediction.
We conduct experiments on various recently released Multi-Event VideoQA datasets, STAR 42
AGQA 14), EgoTTaskQA [66] and NEXT-QA [4]] 'The experimental results demonstrate that proposed Glance-Focus model achieves state-of-the-an performance on all four datasets. We further verify the effectiveness of our approach via model analysis, showing the outstanding performance of key components in our method. Moreover, both quantitative and qualitative results affirm that our proposed memory prompts play a significant anchor role in multi- event reasoning by bridging the gap between questions and videos.
2 Related Works
2.1 Video Question Answering
Early attempts I1 49 51 41 have made progress in improving complex spatial-temporal con- textual information modeling for VideoQA tasks. They usually extract the appearance and motion features of video units (such as clips or frames) separately and then let them interact with the question to find an answer. To capture cnestion-related spatial and temporal clues, co-memory and co-attention mechanisms are proposed by II and HME[6] respectively. B2A[35] uses questions as a bridge connect the appearance and motion features to find the answers. These holistic models effective at aggregating spatial and temporal context, making them well-suited for general one-step video reasoning tasks. However, they tend to encounter challenges when confronted with more intricate compositional questions[[4]
Therefore many studies (23 13 30
45) have delved into the exploration of hierarchical modeling strategies to address the challenges of multi-grained video content representation. HCRN proposes using reusable units stacked hierarchically to gradually extract frame-level to video-level features conditioned by motion and question. Graph-based methods such as L-GCN[15). HAIR (30, and VGT utilize extra object detection models, like Faster R- R-CNNB71, to explicitly incorporate object-level features into the model. They further organize the objects into a unified graph for fine-grained reasoning. Although these hierarchical structures contain multi-grained clues, few of them explicitly model event-level structure of videos. This limitation poses a challenge in effectively handling high-level tasks, including event prediction, explanation, and counterfactual reasoning.
2.2 Video-Languag Pre-trained Models
Recently, Transformers- based large video-language models (VLMs) 52 25
50
have accomplished comparable or superior results on downstream tasks like VideoQA. These models harness the power of large-scale image or video-text pairs collected from the Internet to pre-train cross-modal correspondence models. They commonly employ proxy tasks like masked language
modeling and masked frame modeling [25] 50 40) or contrastive learning 47 during pre-training.
Some models like ActBERT (2) and FridgeFormer [12] focus multi-grained semantics grounding by explicitly learning the alignment of objects and actions across modalities. However, applying these VLMs directly to Multi-Event VideoQA tasks is non-trivial. These models often necessitate sparse sampling of lengthy videos, leading to a notable loss of video semantics, and thus suffer from weak correspondence between questions involving high-level event concepts and intricate video content [6]. To tackle this challenge, some retrieval- based cross mrosl methods [48] 19 36 have been proposed recently efficiently extract the keyframes of the video for reasoning- Nonetheless, the use of fixed keyframes may pose difficulties when dealing with multi-grained questions. Instead, we propose employ the dynamic and adaptable event memories for representing lengthy videos.
2.3 Multi-Event Modeling
Multi-event modeling is an essential ability for household robots assist humans in completing complex tasks. However, explicit multi-event modeling still needs to be explored for VideoQA task. TSEA 9 proposes a heuristic method to extract event-level features of videos. Several studies targeting at multi-activity recognition and detection task [5, 34 33
employ graph structures to model complex relationships among multiple activities. P-GCN 15 extracts activity proposals from videos and utilizes graph models to integrate context. Top organizes egocentric videos into several
3

---

data flow attention
A
A. put down
Transformer Decoder
Bundwoud Anowau
Transformer Decoder
memory query
() focus on memory
query
b) focus
Transformer Encoder
Transformer Encoder
0-00 what the dlothes?
N-2
N-1
1.Glancing Stage
2.Focusing Stage
Figure The complex multi-event reasoning task is structured into two stages: Glancing and Focusing. At the Glancing stage, a set of unordered event memories are obtained by aggregating relevant input video frames. At the Focusing stage, these event memories are reorganized according to their sequential order. The resulting memory prompts act as bridges, allowing model progressively focus on the related video frames for answer prediction using multi-level cross-attention mechanism.
zones and learns the links between these zones for scene affordances and future action prediction. Bridge-prompt22 (2 leverages the continuity the natural language to concatenate multiple discrete events by constructing a series of text prompts. Similar to the multi-activity detection task, our goal is to obtain set high-level key event from the video. However, directly training an action detection model requires additional event-level annotations, which may not always be available. Furthermore, impractical use a closed set of actions to encompass all possible events in the real-world. By contrast, our proposed approach learns dynamic event memories from videos through either unsupervised generation or supervised set-matching methods, depending on the availability event labels during the training process.
3 Approach: The Glance-Focus Model
We first give an overview of our model in Section 3.1 Next, we describe in detail the two stages of our model, the Glancing stage and the Focusing stage in Section 3.2 and Section 3.3 respectively. Finally, in Section 3.4 we illustrate the overall loss functions.
3.1 Overview
Given video clip V and question g, VideoQA model predicts the answer y, formulated as follows:
i = arg max Folgg, V.A)..
(1)
where is the predicted answer chosen from the candidate answers denoted as A, i.e., answer vocabulary for open-ended questions choices for multi-choice questions. 0 is the trainable parameters of a VideoQA model
We propose novel Glance-Focus VideoQA model with the Transformer Encoder-Decoder framework(39) [39] As shown Figure 2 the model answers the question with two stages: 1) At the Glancing stage, set of episode memories are queried from the input video; 2) At the focusing stage, the model reorganizes the memories into memory prompts and they will serve as anchors to localize the question-related video content by multi-level focusing.
Input representations. For the input multi-event video of length T, we use pretrained visual backbone to extract its features X RTXD with dimension D. The video features are fed into the model both glancing and focusing stages. For questions, following [II, we tokenize them using Byte-Pair Encoding (BPE) tokenization, and embed them with pretrained RoBERTa language

---

m3del[31] to get textual tokens w = fwi..... wo} with question length L. Then they are projected into the same dimension feature space video features with a projector g(.) get question embeddings L = ((w) RRXDD The question embeddings are fed into the model only at focusing stage.
3.2 Glancing Stage
It is challenging for the VideoQA model to directly find answers from lengthy videos containing multiple complex events. To efficiently establish the corresponding relationships between the high- level event concepts of the questions and the low-level video unit features, we propose employing quick glance strategy that effectively extracts high-level episode memories from the video to guide the subsequent reasoning.
To obtain the episode memory, a direct approach would be training an action detection model to extract a set of actions from the video. However, this may not be applicable in certain cases where video data lacks event annotations. Moreover, it is impractical world with a close-set vocabulary. To address these issues, we propose more flexible and practical solution which can automatically learn a set of dynamic memories M = {mı,... mN) E R x to summarize the key N events occurring in the videos. Given a group of randomly initialized memory queries, we train a Transformer cncoder-dccodee((3) to update them by gradually aggregating the related encoded video features X through cross cross-attentnes attention interactions. Specifically, we design both unsupervised generation and supervised matching approaches depending on whether event-level annotations are available.
encompass all possible events in the real
Unsupervised event memory generation. In this case, we employ the idea of information maximization 21 29 from the unsupervised learning to generate set of event memories. Our goal is to generate collection of event memories. To ensure that this set comprehensively captures all the crucial events within the video, we impose two vital constraints. 1) Individual Certainty. Each individual event encouraged be distinctive, thereby discouraging individual frames from forming separate and independent events. 2) Global Diversity. We aim to maintain diversity within the event set, preventing all video frames from being clustered into only one single event.
Assuming that there are C' types of events that will occur in the video. For individual certainty, we suppose higher classification confidence score. Namely, the ideal category distribution is similar to one-hot encoding 21]. Specifically, a classifer header fe of a linear layer with softmax is used to predict the event classification logits P... = fe(m). The individual certainty loss is denoted as
Loers (foi M) = -EMEM
Σ p'n log Pi.. 1=1
(2)
To ensure global diversity, we consider both the diversity of semantics and temporal spans within the event set. To promote semantic diversity, we assign category labels evenly across all memories. Namely, in contrast to the above one-hot classification logits of cach event, we aim for an even distribution. The following L.s is minimized to constitute the semantic diversity loss
(ats(fe M)
Σ log pt
where p = EEmemEiffe (f))) is the mean output embedding of the all event memories a video sample. pursue reduction in the temporal overlap of all events. We use a temporal header ft of 3-layer FFN with ReLU to predict the normalized temporal center coordinate and width vi of events, f = fr(m) = (,, vo] E [0, 1]2. . The temporal IoU loss formulated as,
Next, ensure temporal span diversity,
Limu (sn M) = EMEM
Σ (o, t,f=1
(4)
where loU(.) is the normalized temporal lo given the paired temporal prediction of events. The overall global diversity loss is the weighted sum of Lole and Liou
5

---

Laso fr;M) detshals + Xion
(5)
Supervised event memory matching. For datasets with ground-truth event-level labels, we similarly use the set prediction via bipartite matching [4[ to extract event memories. Since the output of the Transformer is unordered, we do not have one-to-one correspondence between the predicted event memories and the ground-truth events. Following 4 (24,, we use the Hungarian algorithm to find the optimal bipartite matching results. Given the padding ground-truth event labels E' {++);-)
{ca Ti 16,, wr]), where and are class labels and the temporal labels respectively, the matching cost Cmatch is defined as:
Cmaach
Σ -I--c, 1cs7odee0700 + +es0odelll7 fxcill.
(6)
where π
E Ny is the permutation of N event memories. Then the goal is to find the optimal æ to minimize the matching cost.
Memory prompting. At the glancing stage, the output is of unordered event memories M = "unn,,, ). We reorganize them into M the sequential order according to their temporal center coordinate }6.}. The temporal positional embedding P. E { E [1, NJ} is added to the corresponding events to indicate their temporal positions in the video. And they will serve as the memory prompts during multi-level attention the following focusing stage.
3.3 Focusing Stage
is difficult to directly associate the high-level event semantics in the questions with corresponding low-level videos. Thus the event memories extracted through video glancing serve as useful anchors to prompt the model quickly locate event segments related to the questions. Video features X. memory prompts M + Pt, and question embeddings are concatenated as input. The Transformer Encoder embeds these embeddings from different sources into joint space [Xeney Mencs Lene].
Multi-level cross attentions. Given set of answer queries a E RNNxD, the Transformer Decoder exploits multi-level cross attention to replace the original multi-head cross-attention to guide the model gradually focus on the question-related visual clues the video.
a) Focus on memory. Given questions with high-level event semantics, we first use question embed- dings queries to interact with memory prompts to guide the model focus the corresponding event memories M Moo.
Q Lene, K V Menc
M for Multihead Q K, V).
(
b) Focus on frame. Then the selected event memories prompt the model to localize the corresponding video frames foc as visual clues for specific reasoning:
Q Moc foc: K = X
X foc = Multihead Q, K.
(10)
c) Predict the answer. Finally, the answer queries find the answers å through reasoning across the focused video frames:
Q=a.=V-XX- X
(11)
â Multihead Q.
(12)
Similarly to classifier header fc, an answer classifier header fa of a linear layer with softmax is used to predict the answer label from the last token of answer queries y = Sa (a).
3.4 Loss Functions
The first part of loss functions is the answer classification cross-entropy loss Laga. The overall unsupervised loss is defined linear combination:
Luna Fos q; V; A: Lgo + Ncert + Laie.
(13)

---

Under the situation with ground-truth event-level labels, we turn to use them for supervision by minimizing the event classification cross-entropy loss Lds and the temporal span L1 loss Cr. between the matching memories with the ground-truth events:
L (Fo V; A: E) Lga + Aashess + Ar..r.I
(14)
4 Evaluation
4.1 Datasets
We evaluate model various recently proposed Multi-Event VideoQA datasets. The comparison of them is shown [ Note that we use the v2 version of the AGQA dataset, which has more balanced distributions, as the dataset creator recommended. We choose these benchmarks for broad coverage of: Video content. Including household and general activities, etc. Video duration. The average video length ranges from 12 seconds to 44 seconds. Event complexity. All these videos contain multiple events or actions, with the average event number ranging from 2.7 to 8.8. Event complexity. The number of event categories varies from 50 793.
Table 1: Comparison of the Multi-Event VideoQA benchmarks.
Datasets
Avg. Events/Video
Video Length
#Event Classes
#Videos
QAs
STAR 42
2.7
12s
157
22K
60K
AGQA 14
6.8
30s
157
9.7K
2,27M
25s
793
2K
40K
NEXT-QA(43)
44s
50
5.4K
52K
4.2 Implementation Details
For each benchmark, we follow standard protocols outlined by prior works for data processing, metrics, and settings ensure fair comparisons. For video feature extraction, we use frozen S3D model [46] pre- trained on How HowTol00M on STAR dataset and C3D[38] feature on Ea dataset following [6] We also try Faster-RCM, model pretrained on the Visual Genome [22] and CLIP(VIT-B/32) for more challenging AGQA and NExT-QA datasets. We employ standard 2-layer, 8-head Transformer Ennoder-Decoder with hidden size D of 512 as the backbone for our Mlance-Focus model. According to the statisties of common event categories, the query number N is set to 10 since the maximum average number of events per video in the current dataset is approximately 8.8. And the event class number C is set the ground truth overall class number in the dataset or 100 if it is unknown by default. For training details, we use dropout of 0.1, and initialize model weights using Xavier init(13) Adam optimizer 20 is used with learning rate of 5e-6 to optimize model parameters. All experiments are conducted on an NVIDIA GeForce RTX 3090Ti GPU. More details can be found in the supplementary material.
4.3 Comparison with State-of-the-arts
We conduct a comprehensive comparison with current sfate-of-the arts (SOTA) VideoQA methods, including traditional spatial-temporal models and large vision language models. We compare our Glance-Foeus model (shorted as GF model in the following) with them on all four datasets. The results are shown in Table 2 3 and resppective. Overall, our proposed GF model achieves SOTA results on all benchmarks, and obtains significant advantages in both the STAR and the Ego TaskQA dataset. We will analyze different datasets one by one follows. Note that in Table the red numbers highlight the significant gains compared to prior SOTA results.
For STAR dataset, our proposed model obtains excellent reasoning performance shown in Table 2 When facing all four different types of questions, GF significantly surpasses the previous SOTAS. especially on the Sequence category. Such results reflect that our model has strong temporal contextual modeling ability. What's more, note that our model trained without ground-truth event labels, i.e., GF(uns), has achieved comparable results with the one trained under supervision, i.e., GF(sup). It indicates that our proposed unsupervised memory generation in the Glance stage can effectively extract key events with strong discriminability.
7

---

Table 2: QA accuracies of SOTA methods on STAR test set (* indicates the val set results).
Model
Interaction
Sequence
Prediction
Feasibility
Mean
25
39.81
43.59
32.34
31 31.42
36.70
RESERVE-B 50
44.80
42.40
38.80
36.20
40.50
Flamingo-9BD
43.40
*
47.53
50.81
47.75
44.08
47.54
Temp|ATP| A
50.63
$2.87
49.36
40.61
48.37
10]
55.59
$4,23
54.24
44.48
51.13
GF(uns)
51.91
63.06.-a0
54.89
45.57
53.80-2213
(i((su))
56.10
61.27Gnan
$2.65
45.74
53.94
We also evaluate our model in the normal split of the EgoTaskQA dataset. From Table 3 we can find that our model outperforms SOTA methods on almost all question types. Our unsupervised GF model has strong performance especially on the intent and explanatory question types. These questions evaluate the understanding of the humans daily activities, which are all necessary for building the household robots. And the model trained under supervision achieves strong performance especially object and predictive question types, which indicates that by explicitly modeling the events happening in the video, the model can obtain stronger event semantic understanding ability, namely, recognize the interacted objects and to predict the next action.
Table QA accuracies of SOTA methods on EgoTaskQA normal split.
Category VisualBERT 26
ClipBERT 25
HCRN 23
GF(uns)
GF(sup)
world
39.73
42.15
44.27
44.65
46.53
Scope
intent
44.51
40.94
49.77
52.07e230
51.88
multi-agent
26.29
27.63
31.36
33.17
34.31
descriptive
41.99
38.45
43.48
44.48
46.68.1..0
predictive
30.37
31.50
36.56
39.00
40.36.+1.0)
Type
counterfactual
41.99
48.00
46.80
explanatory
37.42
42.39
40.60
42.45
42.18
action
15.02
22.91
14.92
15.68
15.68
object
23.26
21.80
45.31
45.97
50.98 (5,67)
Semantic
state
59.20
54.36
68,28
70.44
71.23
change
68.27
66.58
67.38
68.69
69.71
open
24.62
27.70
30.23
32.79 (2.56)
Overall
binary
68.08
67.52
69.42
70.25
37.93
39.87
42.20
43.06
44.27
As for AGQA dataset, from Table 4 our model also achieves the SOTA result with object-level video features, i.e., GF(sup) with Faster R- R-CNN(77 backbone. And the one trained with S3D features also gains comparable results. We show several representative reasoning types of the dataset. Our model obtains better results on some challenging question types, such as obj-rel composition questions and temporal superlative questions like "the longest...", Overall, our model shows more clear advantage over SOTA on the challenging Open questions, compared to the relatively simple Binary questions for which the model could gain a 50% accuracy rate by blind guessing. The complete breakdown results on all kinds of question types and a more thorough analysis can be found in the Supplementary Material.
Table 4: QA accuracies of SOTA methods AGQA v2 test
Method
6
HCRN 23
A10 ViT
Temp CLIP
MIST(IO) CL.IP
GP((ns)
GF(uns)
GF(sup)
Backhone
C3D
C3D
S3D
FasterRCNN
FasterRCNN
obj-rel
37.42
40.33
48.34
50.15
51.68
52.93
54.31
54,96.339
superlative
33.21
33.55
37.53
39,78
42.05
42,78
42.90
44,62 (255)
sequencing
49.77
49.70
49.61
48.25
67.24
53.03
51.97
53.24
exist
49.96
50.01
50.81
51.79
60.33
58.31
58.08
59.13
duration
47.03
43.84
45.36
49.59
54.62
50.86
52.02
52.80
recog.
5.43
18.97
18.96
19.69
22,08 2.3
16.38
14.17
open
31.01
50.56
53.06
55.66
56.07
binary
48.91
47.97
58.28
53.61
53.52
54.17
all
39.89
42.11
48.59
49.79
54.39
53.33
54,59
55.08

---

We also evaluate our model on the challenging NExT-QA dataset based on different video backbones. Note that the NExT-QA dataset does not provide event annotations. Therefore, we exploit the proposed unsupervised memory generation in the Glance stage. As shown in Table [] without event-level supervision or applying any extra action detection model, the unsupervised GF models with the same backbone other methods still achieve SOTA performances. Such results further justify the strong multi-event reasoning ability of our method, considering that NExT-QA is currently the most complex Multi-Event VideoQA benchmark with longer video duration (44s) and richer events (8.8 actions per video).
What's more, we also report the breakdown results by the different question types. According to the results, GF has strong event reasoning ability especially on Descriptive (Acc@D) D) questions. improving the accuracy by 3.61% 6,1,, and also on event relationship reasoning, achieving 2.31 % accuracy gain on Causal (Ace@C) questions. As recommended ATP we additionally report results on its proposed ATP-hard subset of the NEXT-QA validation set. This subset filters out those "easy" questions that can be answered with single frame. The results in Table show that our GF models significantly surpass HGA[17 models this subset by improving 5.22% accuracy overall. The above results indicate that compared to HGA and other methods that directly model the correspondence between question and long video through co-attention, our method using event memory as an intermediate bridge to establish such correlation for reasoning is more efficient.
Table 5: Accuracy (%) comparison on NExT-QA validation set and the ATP- subset 3 Acc@C, D. denote accuracy for Causal, Temporal, and Descriptive questions respectively.
Methods
Backbone
val
(ATP-hard subset)
Acc@C
Tcc@T
Acc D
Acc All
Acc@C
Tcc@T
Acc All
ATP
CLIP
48.30
49.30
65.00
49.20
19.60
22.60
20.20
VQA-TI 27
S3D
49.60
51.49
63.19
$2.32
IGV 28
C3D
48.56
51.67
59.64
$1.34
ATP
CLIP
53.10
50.20
66.80
54,30
38.40
36.50
38.80
HGA
C3D
46.26
50.74
$9.33
49.74
43.30
45.30
44.10
MIST 10
CLIP
54.62
56.64
66.92
57.18
GF(uns)
S3D
51.78
54.71
63.96
54.62
48.28
49.95
48.96
GF(uns)
C3D
53.59
55.71
56.33
48.50
49.52
GF(uns)
CLIP
56.93
57.07
(3653
58.83
48.65
50.27
49.32-5m
4.4 Model Analysis
In this section, evaluate the effectiveness of our proposed modules by comparing different variants of the GF models and various baselines. We first introduce the baselines and GF variants as follows. Baselines: 1) MDETR which utilizes same 2-layer Transformer Encoder-Decorde architecture. 2) An Oracle model with ground-truth event labels concatenated to the questions as textual input as an upper-bound bound model. GF Variants: 1) Glance-only. Model which uses standard cross-ates -atsstattttionn to replace the multi-level attention strategy. 2) Detect-Focus. Apply a frozen action detection model, such SlowFast to replace the Glance stage of the model. We try to keep the model parameters consistent.
Effect of memory prompting. Firstly, the performance of the Oracle model significantly improves compared to the baseline model. convinced us that event memories do play an important prompting role in the Multi-Event VideoQA task. Besides, it can be found that with event memory generation at the glancing stage, the Glance-only model outperforms the MDETR baseline and the model using frozen action detection (Detect-Focus) with large gap both datasets. Such results reflect the effectiveness of our proposed eemoryyproopting strategy. Compared to additionally fine- sin-ttnnng an action detection model, our proposed Glance strategy can adaptively generate key event/action features of the long videos without any annotations.
What's 's more, we evaluate the effectiveness of different loss functions that are designed during the event memory extraction Table We carefully evaluate all pairs of losses on STAR, EgoTaskQA and AGQA datasets. Firstly, the model with complete losses achieves the best results on both unsupervised and supervised settings of datasets. indicates that each loss plays its role in generating the event memories. Secondly, under the unsupervised setting, the Liou shows

---

important role in controlling the temporal prediction of the events. For supervised settings, the Lets shows more important role for the event semantic supervision helps cross-modal grounding. Please see more discussions on the loss functions in the Supplementary Material.
Effect of multi-level attention. From Table 6 compared to the Glance-only model using standard cross-attention, the Glance- Focus model with our designed multi-level attention shows stronger reasoning ability on both STAR and EgoTaskQA datasets. Taking an example from STAR dataset, we further visualize the attention weights in different levels in Figure [3] the darker the
Table 6: Module ablations on STAR and EgoTaskQQ datasets.
Model
STAR
EgoTaskkQ
MDETR
46.26
39.68
color, the greater the weight. The attention map in the first row shows the correspondence of the question embeddings with the Glance-only
Detect-Focus
49.85
42.62
53.12
42.70
generated memories. this level, the model focuses on the most related event "Throwing towel"" The attention map in the sec- ond row is the correspondence between the focused event and the video embeddings. At this level, the model correctly attends to the most related keyframes for answer prediction.
Glance-Focus
53.94
44.27
Oracle(Upppr)
55.19
49.52
Table 7: Ablation study of the loss functions all datasets.
Uns/Sup
Liou
Leert
LL1
AGQA
STAR
EgoTaskkQ
Uns
54.12
$3.62
42.69
Uns
52.89
51.35
41.77
Uns
54.27
53.72
42.85
Uns
54.59
53.86
43.06
Sup
54.26
53.69
43.87
Sup
53.19
53.03
42.64
Sup
55.08
53.94
44,27
levent
Surnece suling so
Fusiin some food son sonhnnu
Weshing somutio with lowal
Tironing fowdls somcehere (0-4290 (2223))
Waocingtooooe Looooii outaide windoow
wothhl
(o 00,00,, n-404)
(0-2008 0-4106)
10
Phrting some nomewhend 00.2964, 0.4277)
ing whhhhhingg 00.8601, 0.2898)
Throwing somewifore (0.46)2.0
phonelcamera (0.7806,
0.3042)
Tirowing
Holding a dish
Putting same food somendoeee
Washinng something with atowel
Throwin wfnwwelys vomershere
Holding phone/camera
towel/s
(Min)
Question: Which object dld the person theow afeer they put dorwn the fond?
Predicted Ansaee: The towel.
Figure 3: Visualization of the event memory generation and the multi-level attention map.
Conclusion
Our proposed Glance-Focus model provides an innovative approach to the challenging Multi-Event VideoQA task. With set of event memories as prompts, the model can efficiently establish the correspondence between high-level questions and low-level video frames. Our experiments has shown that event memories play a crucial role in locating inestion--ellted visual clues for reasoning. The proposed Glance-Focur strategy can obtain a complete and compact representation of videos without additional annotations, which can be extended to more video comprehension tasks, such as video captioningsummmarzztionn We hope this study can inspire more future research the field of Multi-Event Video Understanding.
10

---

Acknowledgements
This work is partially supported by National Key R&D Program of China No. 20212D0111901, and Natural Science Foundation of China under contracts Nos. U21B2025, U19B2036.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 2022.
[2] Max Bain, Arsha Nagrani, Gül Varol. and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 1728-1738, 2021.
[3] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the "video" in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917-2927, 2022.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, pages 213-229, 2020.
[5] Long Hoang Dang, Thao Minh Lc, Vuong Lc, and Truyen Tran. Hicrarchical object-oriented spatio- temporal reasoning for video question answering. International Joint Conference on Artificial Intelligence,
2021.
[6] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heterogencous memory enhanced multimodal attention model for video question answering. In Proceedings of the
IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1999-2007. 2019.
[7] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He.. Slowfast networks for vidco recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202-6211, 2019.
[8] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. Violet: End- to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021.
[9] Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: a video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 1675-1685, 2021.
[10] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. arXiv preprint arXiv:2212.09522 2022.
[11] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6576-6585, 2018.
[12] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridgeformer: Bridging video-text retrieval with multiple choice questions. Proceedings of the IEEL/CVF Conference on Computer
Vision and Pattern Recognition, 2022.
[13] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[14] Madeleine Grunde-MeL MeLaughinn Ranjay Krishna, and Maneesh Agrawala. Agga: A benchmark for reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287-11297, 2021.
compositional
[15] Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan, Location-aware graph convolutional networks for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1 11021-1100 11028, 2020.
[16] Baoxiong Jia, Ting Iei,, Loon-CChun22 Zhu, and Siyuan Huang. Egotaskqu: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 2022.
[17] Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11 11109-11116. 2020.
[18] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEF/CVF
International Conference on Computer Vision, pages 1780-1790, 2021.
[19] Sungdong Kim, Jin-Hwa Kim, Jiyoung Lee, and Minjoon Seo. carameeraettric bideo-grounded text generation. arXiv preprint ai3vv:2.01. /050, 2023.
[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference
Learning Representations, 2015.
[21] Andreas Krause, Pietro Perona, and Ryan Gomes, Discriminative clustering by regularized information maximization. Advances in Neural Information Processing Systems, 23, 2010.
11

---

[22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen. Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crovdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 2017. www.areeecccccrreaeee [23] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9972-9981, 2020.
[24] Jie Lei. Tamara L Berg. and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846-11 1858, 2021. [25] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-langusge
learning via sparse sampling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 7331-7341, 2021.
[26] Liunian Harold Li, Mark Yatskar, Da Yin. Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, arXiv preprint arXiv: NXN::/908.03557 2019.
[27] Muheng Li. Lei Chen, Yueqi Duan. Zhilan Hu, Jianjiang Feng, Jie Zhou, and Jiwen Lu. prodgee-rompt: Towards ordinal action understanding in instructional videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19880-19889, 2022.
[28] Yicong Li, Xiang Wang. Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2928-2937, 2022.
[29] Jian Liang. Dapeng Hu. and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039, 2020.
[30] Fei Liu, Jing Liu, Weining Wang, and Hanqing Lu. Hair: Hierarchical visual-semantic relational reasoning for video question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv
preprint arXiv: (ttf:::997.11166100 1692. 2019.
[32] Antoine Miech, Dimitri Zhukov,
Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00ms Learning a text-vidco embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640., 2019.
[33] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 163-172.220 2020.
[34] Megha Nawhal and Greg Mori, Activity graph transformer for temporal action localization. arXiv preprint
ar arXiv:2101.088 2021.
[35] Jungin Park, Jiyoung Lec, and Kwanghoon Sohn. Bridge to answer: Structure-aware graph interaction network for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15526-15535, 2021.
[36] Tianwen Qian, Ran Cui, Jingjing Chen, Pai Peng, Xiaowei Guo, and Yu-Gang Jiang. Locate before answering: Answer guided question localization for video question answering. IEEE Transactions on Multimedia. 2023.
[37] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards time object detection with region proposal networks. Advances in Neural Information Processing Systems, pages 91-99, 2015. [38] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 4489-4444 4497. 2015.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems.
30, 2017.
[40] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai. Jianping Wu, Ying Shan, Xiaohu Qic, and Mike Zheng Shou, All in one: Exploring unified video-language pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[41] Janyu Wang. Bing-Kun Kun Bao, and Changsheng Xu. Dualvgr: dual--viuul graph reasoning unit for video question answering. IEEE Transactions on Multimedia, pages 3369-3366 3380, 2021.
[42] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world vidcos. In Advances in Neural Information Processing Systems, 2021.
[43] Junbin Xiao, Xindi Shang, Angela Yao, and Tat- Sat-Seng Chua. Next-qa: Next phase of question- anwwering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2021.
Recognition, pages
[44] Junbin Xiao, Angela Yao, Zhiyuan I. Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video as conditional graph hierarchy for multi-granular question answering. Proceedings of the AAAI Conference on Artificial Intelligence. 2022.
[45] Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. Video graph transformer for video question answering. In Proceedings of the European Conference on Computer Vision, pages 39-58, 2022. [46] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: accuracy trade-off' in video classification. In Proceedings of the European
12

---

Conference on Computer Vision, pages 305-321. 2018.
[47] Antoine Yang, Antoine Miech, Joset Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1686-1697. 2021.
[48] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.0663 2023.
[49] Zhou Yu, Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 628 1-6900, 2019.
[50] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Multimodal neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[51] Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhenxin Xiao, Xiaohui Yan, Jun Yu, Deng Cai, and Fei Wu. Long-form video question answering via dynamic hierarchical reinforced networks. IEEE Transactions on Image Processing, pages 5939-5952 5952, 2019.
[52] Linchao and Yi Yang. Acthert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8746-8755, 2020.
13


Reviews:
[{"review_id": "1C2TQ5Aonr", "reviewer": "Reviewer_veHn", "summary": "This paper addresses the task of question-answering in videos which have multiple events. The authors draw inspiration from the human capability to create event memories as a way to anchor the reasoning process over multiple events, and propose the Glance-Focus model, which aggregates a set of event memories (during a first \"glancing\" stage) and then proceeds to further process these memories (\"focus\" stage) to produce the final answer to the query. The authors benchmark their model on several standard multi-event benchmarks, such as STAR, AGQA (the balanced v2 version), and EgoTaskQA to characterize the effectiveness of their approach.", "questions": "Overall, the rating for this paper pre-rebuttal is borderline. If the authors can address and clarify the points/questions raised in the weaknesses above, that would be helpful for updating the final rating and better grounding the discussion phase with the other reviewers.", "limitations": "The authors don't seem to provide a detailed discussion of limitations in the work, though the societal impacts does seem to be similar to related work in this space. It would be helpful and clarifying if there was an better indicator of how aspects of the proposed technique, e.g. the supervised bipartite matching, scale with longer videos, and how such limitations may be better addressed by future work.\n\n---\n\n**Post-rebuttal update:** The authors largely address some of the key concerns that were brought up in the original review, in particular, the additional analysis on NExT-QA significantly improves the evaluation (the results on ActivityNet are promising, but not the *primary* focus of the updated review given the context and discussion below). While there are some still areas that can still be further improved (as detailed and described in the rebuttal discussion below), I am increasing my rating towards leaning acceptance since I believe that these are areas that can be further addressed by future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "`+` Question-answering in videos is an important task and challenging domain for multimodal models. Further, the goal of temporal reasoning is an important step for the development of AI agents in general.\n\n`+` The general direction of reasoning on top of generated/matched event memories draws inspiration from human event memory/reasoning, and seems like a reasonable approach.\n\n`+` The authors evaluation their technique on several benchmarks (representing different video distributions) and show improvements; I also appreciated that the authors looked at the v2 version of AGQA, for better characterization of the model capabilities.", "weaknesses": "`-` The paper emphasizes \"long video\" understanding, but the length of the videos in the datasets examined are generally quite short (12-30 seconds) relative to many other video QA settings (e.g., ActivityNet-QA is ~2 mins). I understand this may be an inherited term from the prior work, MIST [10], which also uses similar terminology. However, that prior work did at least evaluate on NExT-QA, which is ~50% longer on average than the longest video dataset examined in this work here (AGQA).\n\n`-` Additionally, while the results on the short benchmark (STAR) are significant, the results become significantly more mixed when compared with closely related work (e.g., [10]) on longer benchmarks like AGQAv2 (+0.69 overall, and some specific categories like \"sequencing\", which seems like it should be better with event reasoning, are much worse than prior work). This also brings to question the efficacy of the proposed approach on longer video settings.\n\n`-` Further, the mixed results are even more striking due to the fact that this work leverages stronger low-level event supervision (for the bipartite event memory matching) than prior work (e.g., MIST) does. It is unclear how, without this level of supervision, the gains of the proposed technique, relative to prior work, are impacted (or the limitations of this approach as the videos become longer).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Scale to longer video**\n\nWe thank the reviewer's suggestions. The evaluations on longer VideoQA benchmarks (NExT-QA/ActivityNet-QA) are shown in the [global rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=LQPJMIxMFG). Note that the model is tested under the unsupervised setting since the two datasets both do not have event-level annotations. These results can further indicate the effectiveness of the proposed approach on longer video settings.\n\n**Event Reasoning Ability Analysis**\n\nWe found that [MIST](https://arxiv.org/abs/2212.09522) performs very well on the sequencing questions compared to other works. Specifically, it achieves a 13.88% improvement compared to GF. For further analysis, we calculate the breakdown accuracy results on question types (superlative/sequencing/duration) related to multi-event reasoning ability by the answer types (binary/open-ended). The results in Tab.1 indicate that although MIST performs better than GF in question types such as sequencing and duration, its performance advantages mainly come from the binary answer questions, which may indicate that the model overfits the data bias. Nevertheless, GF performs better on all open-ended questions, which reflects its stronger generalization ability.\n\n**Table 1.  Breakdown results on question types related to event reasoning by the answer types.**\n\n| Methods                                  | Superlative                      | Sequencing                     | Duration                       |\n| ---------------------------------------- | -------------------------------- | ------------------------------ | ------------------------------ |\n| [AIO](https://arxiv.org/abs/2203.07303)  | 37.53                            | 49.61                          | 50.81                          |\n| [ATP](https://arxiv.org/abs/2206.01720)  | 39.78                            | 48.25                          | 51.79                          |\n| [MIST](https://arxiv.org/abs/2212.09522) | *42.05*(B: **56.36**/O: *28.35*)     | **67.24**(B: **67.43**/O: *26.45*) | **60.33**(B: **55.33**/O: 24.59)   |\n| **GF-obj(sup)**                          | **44.62**(B: 56.20/O: **33.62**) | *53.36*(B: 53.35/O: **26.91**) | *52.80*(B: 53.43/O: **25.61**) |\n\nAdditionally, we also report the breakdown results on the longer NExT-QA dataset by the different question types(Causal/Temporal/Descriptive) in Tab.2. According to the results, GF shows a strong temporal reasoning ability than MIST. It indicates that GF still has strong event reasoning ability although the videos become longer.\n\n**Table 2. Evaluation breakdown on NExT-QA by the different question types.**\n\n| Methods                                                      | Acc@Causal | Acc@Temporal     | Acc@Descriptive  |\n| ------------------------------------------------------------ | ---------- | ---------------- | ---------------- |\n| [HGA](https://ojs.aaai.org/index.php/AAAI/article/view/6767) | 46.26      | 50.74            | 59.33            |\n| [Just Ask](https://arxiv.org/abs/2012.00451)                 | 51.4       | 49.6             | 63.1             |\n| MIST                                                         | *56.6*     | *54.6*           | *66.9*           |\n| **GF(uns)**                                                  | **56.93**  | **57.07(+3.07)** | **70.53(+3.63)** |\n\n**Limitation Discussion**\n\nWe have provided discussions on limitations in the supplementary materials. Based on the all reviews, we further summarize and update the limitation discussions here.\n\n- The current videos collected in the Video QA datasets are not long enough and the event is not complex enough to fully evaluate the model's multi-event reasoning ability. We look forward to further progress on multi-event reasoning research. Also, future works including keyframe selection strategies are considered to improve the long video understanding efficiency.\n\n- The unsupervised setting is not as effective as the supervised setting. However, event annotations are expensive. In the future, more efficient and effective event-generation methods will be considered, such as automatically generating pseudo labels for unsupervised training.\n\n- The constraints on the generation of the event memory under unsupervised settings are relatively straightforward. More intuitive ideas from the way how human beings generate event memories are desired to be explored."}, {"review_id": "zdm3xQm46a", "reviewer": "Reviewer_16Tf", "summary": "The paper introduces a model called Glance-Focus for Video Question Answering (VideoQA) tasks, which involve understanding human behaviors in videos. While vision language models have shown success in multi-modal tasks, reasoning over long videos with multiple human-object interaction events remains challenging. Humans can tackle this task effectively by using memory to quickly locate key moments for reasoning. To mimic this reasoning strategy, the Glance-Focus model is proposed.\n\nAt the glancing stage, an Encoder-Decoder is trained to generate dynamic key event memories by aggregating relevant video frames. This is achieved through unsupervised memory generation using individual discriminating and global diversity loss, as well as supervised bipartite matching losses. At the focusing stage, the event memories enable the model to focus on the key event memory and then on the most relevant moment for specific reasoning using multi-level cross-attention. \n\nThe model is evaluated on three representative VideoQA benchmarks (STAR, EgoTaskQA, and AGQA) and achieves state-of-the-art results, surpassing current large models in challenging tasks. The Glance-Focus model addresses the challenge of situation reasoning in VideoQA by generating dynamic event memories and using them to focus on key moments in videos.", "questions": "Refer to the above sections.", "limitations": "The codes should be open-source but it seems the paper doesn't mention", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper's organization is good, it clearly explains the basic ideas. \n\nThe paper proposed to involve memory to cross the gap between the concepts and video content in the QA transformers.\n\nThe related work is organized well and will be better to introduce the recent benchmarks together with the VideoQA.\n\nThe implemented experiments are sufficient and the example visualization is helpful.", "weaknesses": "Firstly, in Table 1, AIO / Temp[ATP] / MIST results appear to be directly cited from [1] but seem to be based on the STAR validation part rather than the test part. Only RESERVE is reported on the STAR test. The dataset partition used for evaluation must be clarified for a meaningful comparison.\n\nIn Section 4.1, please specify the average video length of EgoTaskQA. If available, additional statistics, such as the average number of events/action state changes per dataset, would be beneficial since the proposed method detects state changes in the 'glance' stage. This data will aid in understanding how the method handles scenes with varying event complexity.\n\nFurthermore, the paper lacks qualitative evaluation metrics for action/event/moment detection, such as IoU/mAP. These are essential for analyzing accumulated errors.\n\nThe paper asserts the impracticality of using an action detector/event detector due to the close-set results it produces. However, a qualitative comparison with the proposed method using event/action detectors + rule-based filtering (keywords matching), e.g. Moment-DETR [2] and TubeDETR [3], is absent.\n\nLastly, although the paper discusses multi-event detection, it still chooses multiple keyframes/visual clues in the focusing stages, which has been explored in previous works [4] and [5, 6]. A more comprehensive discussion comparing these related works would be appreciated.\n\nSome typos: Line 233 HowTo100m → HowTo100M.\n\n\n[1] D. Gao et al, MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering\n[2] J. Lei et al, Detecting moments and highlights in videos via natural language queries\n[3] A, Yang et al, TubeDETR: Spatio-Temporal Video Grounding With Transformers\n[4] S. Yu et al, Self-Chained Image-Language Model for Video Localization and Question Answering\n[5] S. Kim, Semi-parametric video-grounded text generation\n[6] T. Qian et al, Locate before answering: Answer guided question localization for video question answering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Corrections on STAR Dataset**\n\nWe are very grateful for the reviewer's reminder. The following are the updated results, with GF in the unsupervised setting achieving SOTA on both the Test and Val parts.\n\n**Table 1. Corrections on STAR evaluations.**\n\n| Model                                         | Test Acc  | Val Acc          |\n| --------------------------------------------- | --------- | ---------------- |\n| [ClipBERT](https://arxiv.org/abs/2102.06183)  | 36.7      | -                |\n| [CLIP](https://arxiv.org/abs/2103.00020)      | 38.0      | -                |\n| [RESERVE](https://arxiv.org/abs/2201.02639)   | 40.5      | -                |\n| [Flamingo](https://arxiv.org/abs/2204.14198)  | 43.4      | 42.23            |\n| [AIO](https://arxiv.org/abs/2203.07303)       | -         | 47.54            |\n| [Temp[ATP]](https://arxiv.org/abs/2206.01720) | -         | 48.37            |\n| [MIST](https://arxiv.org/abs/2212.09522)      | -         | *51.13*          |\n| **GF(uns)**                                   | **57.82** | **53.86(+2.73)** |\n\n**Detailed Dataset Analysis**\n\nWe really appreciate the suggestions. The in-depth analysis of the datasets is presented in [global rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=LQPJMIxMFG). Our proposed method performs well on datasets with various average event numbers per video from 2.7(STAR) to 8.8(NExT-QA). Compared to current VideoQA models, the proposed Glance-Focus frameworks can better handle complex multi-event video understanding scenarios. Besides, the method also exhibits robust performance on videos covering different action categories from 50(NExT-QA) to 793(EgoTaskQA). See also in [Rebuttal]() with **Reviewer sLiX**.\n\n**Effectiveness Analysis of the Glance Stage** \n\nSince our proposed event memory is an implicit representation of videos, it is difficult to analyze the accuracy of event generation, especially in the absence of event annotation. Therefore, only quantitative visualization experiments were conducted (Fig. 3/4 in the main paper, Fig. 2 in the supplementary materials). \n\nBased on the suggestions of the reviewer, we further conduct event accuracy evaluations on the STAR/AGQA. We evaluate on the [Charades](https://arxiv.org/abs/1604.01753) dataset, which is the video source of these two datasets, and use the standard [action detection setting](https://prior.allenai.org/projects/data/charades). For a fair comparison, we report the prediction of the Glance Chain from GF trained with the same action supervision. The results are shown in Tab. 2. GF model shows comparable performance with other methods, which indicates that the Glance chain of our model can effectively capture events (actions) in the video.\n\n**Table 2. Action Detection Comparisons with SOTA Methods on Charades Benchmark**\n\n| Method                                                       | mAP  |\n| ------------------------------------------------------------ | ---- |\n| [Coarse-Fine](https://arxiv.org/abs/2103.01302) | 25.1 |\n| [PDAN](https://paperswithcode.com/paper/pdan-pyramid-dilated-attention-network-for) | 26.5 |\n| GF(sup)                                                      | 23.7 |\n\n**Ablation Study of Using Action Detection Model**\n\nThanks for motivating us to conduct such a necessary ablation study. As the reviewer suggested, we try to use [SlowFast](https://arxiv.org/abs/1812.03982), a popular action detection model, to replace our Glance chain and handle the open-set issue by keyword mapping. [Moment-DETR](https://arxiv.org/pdf/2107.09609.pdf) and [Tube-DETR](https://arxiv.org/abs/2203.16434) are more related to our work. However, they aim at a video grounding task with a textual query, which is a different task from action detection/classification. To alleviate training an additional action detection model, we straightly use the SlowFast model trained on the [AVA](https://arxiv.org/pdf/1705.08421.pdf) dataset. It can detect actions within a 60-category close-set vocabulary. Then, we map these actions to the actions annotated in the STAR dataset (157 classes overall) by keyword matching. Then, the filtered actions are used as memories and inputted into the Focus chain for QA. \n\n**Table 3. Ablations on the Glance chain. The results are mean accuracies on the STAR Test Split.**\n\n| Method                 | Acc   |\n| :--------------------- | ----- |\n| Action Detection-Focus | 49.85 |\n| Glance-Focus(sup)      | 53.94 |\n\nThe performance of such a strategy is not that good. The reasons come from the action category gap across different datasets. As stated in the paper, the events in different scenarios are various and difficult to define at the same time. See more discussions on [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=ucQstYUtv5) with the **Reviewer sLiX**. Therefore, unsupervised event generation is necessary, and our work provides an effective implementation.\n\n**Comparison with Related Works**\n\nThank you for providing the latest related work. We will update the comparison with them in the revision. Similarly, we all utilize the idea of finding the most related video content for QA. But all these works directly use questions as queries to retrieve keyframes, while GF dynamically finds visual clues with memory prompts. Compared to them, GF has the following advantages.\n\n- **More accurate.** Keyframes retrieval suffers from the semantic gap between questions and frames. Memory can bridge the gap for they are middle-level representations.\n- **More adaptable.** GF can extract dynamic and adjustable visual cues across the whole video with the multi-level attention, while the above works can only find clues based on the fixed keyframes.\n- **Extensibility.** GF can obtain a complete compact representation of videos without relying on question queries, so it can be extended to more video comprehension tasks, such as video captioning/dense captioning/summarization.\n\n*P.S. The code will be made public after the paper is accepted.*"}, {"review_id": "C3DDfogLKK", "reviewer": "Reviewer_izc9", "summary": "This paper aims to tackle the problem of multi-event VideoQA that involves multiple human-object interaction events. Inspired by the human's reasoning ability, this paper proposes the Glance-Focus model that can reason from coarse to fine. At the glancing stage, the Glance-Focus model can generate some dynamic key event memories to summarize the key events in the video. At the focusing stage, the Glance-Focus model can reason based on the most relevant moment through the cross-attention mechanism. The contributions of this paper are summarized as following:\n(1) This paper proposes a glance-and-focus mechanism to reason multi-event VideoQA in a human-like manner.\n(2) The Glance-Focus model achieves the state-of-the-art performance on some multi-event VideoQA dataset.\n", "questions": "(1) The Glance-Focus model can not only tackle the problem of multi-event VideoQA but also tackle other kinds of reasoning problems. What about its scalability? The authors should provide more analyses.\n\n(2) In the unsupervised manner at the glancing stage, how to ensure the sequential order of events? ", "limitations": "Yes, the authors have discussed the limitations in the supplementary material.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The idea of glancing and focusing on multi-event VideoQA is interesting and intuitive. This human-like design brings great potential for solving such multi-event VideoQA problem. The glancing can be seen as a kind of information compression to anchor key events, which may guide the focusing stage to conduct the fine-grained reasoning.\n\n(2) The motivation of unsupervised event generation at the glancing stage is clear. Because of the open-vocabulary world, it is hard to annotate all the events in the long-time video. To tackle this problem, this paper proposes to enforce the unsupervised constraints to generate meaningful key events.\n\n(3) To generate useful event memory in an unsupervised manner, this paper proposes to enforce the individual discrimination and the global diversity, which can guarantee to learn more meaningful key events.\n\n(4) This paper innovatively proposes to use event memories as anchors to prompt the model to locate the most relevant segments corresponding to the questions. Besides, the multi-level cross-attention mechanism can promote the event memory to focus on the question-related visual clues in a fine-grained manner.\n\n(5) Extensive experiments and analyses show the effectiveness and reasonability of the method. Specifically, the visualization of Figure 4 clearly shows the relationship between the event memory generation and the fine-grained focusing.", "weaknesses": "(1) At the glancing stage, this paper utilizes the unsupervised learning solution to generate meaningful event memories. However, this paper lacks of the analyses about the inherent relations between the unsupervised learning (such as instance discriminative learning) and the key event memory generation.\n\n(2) The formulation of the temporal IoU loss (Eq.(4)) is not very clear and it should be explained more detailedly.\n\n(3) In experiments, some unsupervised results even outperform the supervised manner but necessary analyses about this phenomenon are lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very glad to see all the positive reviews you given. Hope we can address your questions below.\n\n**Relations with unsupervised learning**\n\nWe will add more relevant explanations in the revised version. Similar to individual discriminative learning, we aim to generate a set of event-level individuals with strong discrimination. The unsupervised individual discriminative learning usually sets constraints on the pairwise sample features to learn discriminative instance features representations [[1]](https://arxiv.org/pdf/1805.01978.pdf)[[2]](https://arxiv.org/abs/1904.03436). For instance, maximize the probability of the augmented instance $\\hat{x}_i$ being classified as instance $x_i$:\n\n$$\nP(x_i|\\hat{x}_i)=\\frac{\\exp(f_i^T\\hat{f}_j/\\tau)}{\\sum\\exp(f_k^T\\hat{f}_i/\\tau)}.\n$$\nTherefore, these methods need to construct such sample pairs, which is inefficient. Besides, the quality of the constructed pairs will impact influence the discrimination of the representations. \n\nInstead of focusing on the feature space, we subtly utilize the prior knowledge of the event memories, i.e., the events happening in the video are usually different, different events should have small temporal overlaps, etc. According to them, we directly set constraints on their classification distribution and the predicted timestamps. Among them, we use information maximization to design the individual certainty loss $L_{cert}$ and semantic diversity loss $L_{cls}$ in Eq.(2) and (3). \n\n**Details of Event Timestamp Prediction**\n\nFor the temporal IoU loss, we sum all normalized temporal IoU of each pair of events in a video. The normalized temporal IoU is denoted as: \n$$\ntIoU(\\tau_i, \\tau_j)= \\frac{\\max(0,  \\max(t^b_i, t^b_j)-\\min(t^e_i, t^e_j)}{\\min(t^b_i, t^b_j)-\\max(t^e_i, t^e_j)))},\n$$\nwhere the $[t^b,t^e]$ is the timestamp, i.e., the beginning and end time of the event converted from the temporal prediction $\\tau$.\n\nSince we can predict the timestamp of each event, we can arrange these events in their sequential order. Without class labels, we cannot do one-to-one matching in a supervised manner, but we can ensure events happened in sequential order.\n\n**Unsupervised>Supervised Analysis**\n\nWe try to explain such an interesting phenomenon. \n\n- Firstly, we think such a phenomenon is reasonable for we did not give very strong memory supervision on the model. The GF model is trained with the combined memory loss and QA loss in Eq.(14). And we set the coefficient of memory loss smaller than QA loss, for QA is the final task we want to handle. \n- Secondly, we think such a phenomenon is acceptable since we do not expect event memory strictly grounding to the event annotations. As our human beings generate abstract memories of the video, we expect dynamically adjustable event memories. Therefore, the memories with weaker supervision may become more adaptable in the reasoning stage.\n\n**Scalability to Other Video Understanding Tasks**\n\nAs discussed in [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=zdm3xQm46a) with **Reviewer 16Tf**, our GF model is easy to extend to other video understanding tasks, such as video captioning/dense captioning/summarization. We will evaluate these tasks in future work. Besides, the Glance chain of our model can be used as a strong action detection model as well, which is evaluated in this [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=t7xsOOjmQ1)."}, {"review_id": "dPUKw93v1V", "reviewer": "Reviewer_xLg6", "summary": "This paper proposes a two-stage method of constructing memories of periodical summarization of the video, then using memory prompts to local key frames for reasoning.", "questions": "The results seem not convincing in terms of the proposed focus chain. Would this accuracy improvement come from the additional parameters?\n\nHow is the positional embedding \\phi formulated in Memory prompting? If it's from bi-partite assignment, is it trainable ? \n\nNot clear how IoU is in (4). Can you provide more details. \n", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The SOTA results on three Ego QA datasets.\n2. The hierarchical construction of video abstraction might be a new way of tackling lengthy videos.\n3. The IoU based un-unsupervised learning could be extended to other senarios.", "weaknesses": "The writing needs improvement. Sentences are generally long and hard to understand. \nE.g., typo in abstract \"abtain\".  Long sentence such as line 13-17. \n\n\nThe idea of one-to-one matching of discrete events with memories seem to be outdated. Why not use soft-attention to assign soft weights of each event to the memory. This seems to be a well-studied technique.\n\n\nThe dependency in a)-c) focus is not justified. Would this chain of dependency bring in heavy loss if M_enc is not accurate. The un-supervised construction of heuristic M seems to be not a suitable prompt. What would (9) be Q=M_foc + L_enc? Any ablation about this.\n\nThe results seem not convincing in terms of the proposed focus chain. Would this accuracy improvement come from the additional parameters?\n\nSeems to be not a fair comparison with non-memory methods. Would methods like MIST. What would a baseline that does not construct M_enc explicitly but implicitly construct a M_foc do. Does the additional steps of constraining memory M from Eq2-5 harm the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We will certainly improve our writing in the revised version.* \n\n**One-To-One Matching & Soft-Attention**\n\nThe one-to-one matching is only used under the supervised setting following the recently proposed series of detection works [DETR](https://arxiv.org/abs/2005.12872)/[MDETR](https://arxiv.org/abs/2104.12763)/[Moment-DETR](https://arxiv.org/pdf/2107.09609.pdf)/[Tube-DETR](https://arxiv.org/abs/2203.16434) etc. The authors take it as a simple implementation of set matching. Many other set-matching strategies are well-studied. The soft attention suggested by the reviewer can also solve this problem by training an attention weight, which may somewhat involve additional parameters. But we still looking forward to any better strategies.\n\n**Analysis of the Focusing Chain**\n\nThanks for motivating us to study on the rationality and effectiveness of our proposed multi-level cross-attention. In our implementation, we use standard cross-attention in Transformer at each level of attention. To the authors' understanding, each level of cross attention is a weighted summary of the value $V$ (focusing) with the weight ($W^QQ^TW^KK$) reflecting the relationships between query $Q$ and key $K$. Therefore, it can realize our purpose of gradually focusing on the related visual clues from the video. \n\n**Ablation study of the Focusing Strategy**\n\nAs the reviewer suggested, we ablate the strategy by reserving the question encoding when focusing on the frame. It seems like a good strategy to handle the inaccurate $M_{enc}$ issue. The accuracies on Test split of STAR dataset are shown in Tab.1. However, the performance of the suggested strategy is slightly weaker. \n\nThe reasons may become the reserved question encoding may cause mixed encoding for the model, which may harm the original focus strategy. Nevertheless, before the cross-attention layer, the Decoder has self-attention layers to aggregate information across different inputs. Thus the $M_{enc}$ generated by the Glance chain is relatively effective in both unsupervised and supervised settings.\n\n**Table 1. Ablation study of the focusing strategy on STAR.**\n\n| Method                | #Parameters(MB)       | Uns       | Sup       |\n| --------------------- | --------------------- | --------- | --------- |\n| $Q=X_{foc}$(Original) | 141.427618            | **53.86** | **53.94** |\n| $Q=X_{foc}+L_{enc}$   | 141.427618            | 53.27     | 53.77     |\n| Glance-only           | 141.423522(-0.004096) | -         | 53.12     |\n| Focus-only            | 141.427618            | 52.89     | -         |\n\n**Effectiveness Analysis of the Focusing Chain** \n\nThe focusing strategy will not involve additional parameters. Here, we give a detailed implementation of the Glance-only we used. Specifically, after the glance module generates the memory prompts, we concatenate them with video features as an overall visual embedding and use the standard cross-attention between visual and textual embedding to output the answer predictions. Therefore, the parameters are almost consistent. See parameters comparisons in Tab.1. \n\nTherefore, the performance gain mainly comes from our designed focusing strategy. Due to the strategy being simple but efficient, the accuracy improvement seems relatively weak.\n\n**More Baseline Implementations**\n\nAs the reviewer suggested, we also implement baseline (Focus-only) with no constraints on $M_{enc}$, but keep the focusing chain remained. The results show that our designed unsupervised loss functions depend on the quality of the generated event memories.\n\n**QAs**\n\nAbout the positional embedding $P_t$, it is calculated by a learnable function $\\Phi(.)$. About the temporal IoU loss, please see detailed explanations in [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=C3DDfogLKK) with **Reviewer izc9**."}, {"review_id": "ucQstYUtv5", "reviewer": "Reviewer_sLiX", "summary": "The paper proposes a novel approach for multi-event video question answering (VideoQA) that mimics the human reasoning strategy of glancing and focusing. The proposed method consists of two stages: glancing and focusing. In the glancing stage, the model generates a set of episode memories that summarize the key events in the video and supports both unsupervised and supervised methods, depending on the availability of event-level annotations. In the focusing stage, it uses the episode memories as memory prompts to guide the multi-level cross-attention between questions and videos, and predicts the answer based on the focused video frames. The paper evaluates the model on three challenging multi-event VideoQA benchmarks: STAR, AGQA, and EgoTaskQA. The experimental results show that it achieves state-of-the-art performance on all datasets, and outperforms current large vision-language models in various complex reasoning tasks. The paper also conducts ablation studies and visualizations to demonstrate the effectiveness of the proposed modules and memory prompts.", "questions": "please refer to weaknesses.", "limitations": "The limitations have been well discussed in supplementary materials. The potential negative societal impact is recommended to be addressed as well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method simulates the reasoning process exhibited by humans, namely an initial overview followed by a concentrated examination, thus presenting an interesting and novel approach.\n2. The implementation is flexible and practical, enabling the simultaneous processing of annotated and unannotated videos without dependence on predetermined action vocabularies or event-level annotations.\n3. This approach represents the pioneering effort to incorporate information maximization as a guiding principle for the training of unlabeled data within the realm of VideoQA.\n4. It designs a multi-level cross-attention mechanism that leverages the episode memories as prompts to guide the model to focus on the relevant video frames for answer prediction.\n5. The performance is quite impressive, as it even surpasses current large vision-language models in various scenarios.", "weaknesses": "1. The writing has a large room to be improved.\n2. In L166, the proposed method assumes there are C events in a video. However, it remains unclear whether C keeps consistent across all videos or dynamically varies for every single one. If C is predetermined, it would be valuable to conduct an ablative study to assess the impact of different choices for C on the overall performance of the method.\n3. For unsupervised event generation, the authors impose a strong bias that the category labels should be evenly distributed across all memories. Further elaboration is essential to explain its reasonability. As from a subjective standpoint, only a limited subset of categories can manifest in a video given the contextual constraints (e.g., playing football on the ocean is impossible).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We will certainly improve our writing in the revised version.* \n\nWe thank for the reviewer motivating us to make a detailed analysis of how the event categories influence the model's performance.\n\n**Comparisons of the overall Number of Event Categories Among Datasets**\n\n We first compare the overall event categories numbers among various datasets in the [Global Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=LQPJMIxMFG). Since the source videos are collected from different scenarios, the richness of video events in different datasets can vary from 50 to 793. But on average, there are approximately 100 common categories of events, such as those used in the STAR and AGQA datasets. However, some datasets like EgoTaskQA follow the event label annotations from LEMMA. It uses fine-grained labels which are the composition of a verb, target, etc. part of the events, therefore resulting in approximately 800 categories. \n\nIn summary, the event tag itself is not easy to define, so considering this, we also propose an unsupervised setting that does not rely on action detection.\n\n**Ablation Study of the \"C\"**\n\n$C$ is a predetermined hyper-parameter in our model, denoting the number of event classes that may occur in the video. By default, we set it to the ground-truth overall event categories number of the dataset. As suggested by the reviewer, we conduct an ablation study to evaluate the impact of different choices for C. We separately set $C=[1,50,100,GT]$ to evaluate. The mean accuracies of the Test split on the STAR dataset are reported.\n\n **Table 1. Ablation study on the number of event categories.**\n\n| C                 | Acc      |\n| :---------------- | --------- |\n| 1                 | 51.68     |\n| 50                | **54.58** |\n| 100               | 53.15     |\n| # GT classes(157) | *53.94*   |\n\nSet $C=1$ will severely affect the performance of the model, while setting the $C$ to a random reasonable number will only slightly affect the performance of the model. Thus, the model is not sensitive to the number of categories, which is a strength to handle the scenarios that the dataset with no event-level annotations.\n\n**Strong Bias Issue**\n\nMost interestingly, the model with $C$ set to GT number only achieves the second best result and the model with $C=50$ obtains the best result. That may somehow confirm the insightful opinion of the reviewer. The bias that the events may happen evenly in each video is too strong. Setting a smaller number will be a better choice.\n\nBesides, taking the contextual constraints into consideration may help to design a more reasonable way to generate event memories. For example, sequentially predicting each part of a compositional event, e.g., \"Verb(Put)--Target(Apple)--Prep(On)--Location(Table)\" could be a better way. Such a study will be done in the future.\n\n**Negative Societal Impact**\n\nSince the model is trained on human-annotated datasets, therefore the model may potentially suffer from data bias. It may cause some social problems. But overall, the potential negative social impact of this work is limited."}, {"review_id": "Z1Ud2yZUAw", "reviewer": "Reviewer_yVqG", "summary": "This paper presents a model for video question answering called \"Glance-Focus\". The idea is to first \"Glance\" at a video to obtain a set of N dynamic memories, each represented as vectors of dimension D. \n\nThe memories can either be learned in an unsupervised way or a supervised way. In the unsupervised case, the paper uses a collection of losses encouraging the video frames be assigned to different memory cells. In the supervised case, where there are ground-truth event labels that are localized in time, the paper uses bipartite matching to match the dynamic memories with the ground truth event labels, and that forms the basis of the loss.\n\nNext, the paper uses an encoder-decoder model. The encoder encodes the video features, memory cells, and question embeddings all together. Then, the decoder starts with a representation of the questio, and cross-attends to the memory cells, then to the video frames, and then to the answer.\n\nThe model is initialized from S3D pretrained on HowTo100M, and learned on STAR, AGQA, and EgoTaskQA. The model seems to do well on all the datasets.\n\n--\nupdate - I still agree with my concerns post-rebuttal and would like to keep my score.", "questions": "What do you see the role of each of the losses? Are all important, and could any be removed? Are there any architecture ablations that could be done while keeping the model size the same?", "limitations": "nope, would be great to add such a section.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The core idea that this paper seems to be tackling is an important one. Videos are long and redundant, and dealing with this space better is an important area of research.\n* The model seems to perform quite well on the STAR test set - (though, it should be noted, it was trained on the raw videos from that dataset, so it might be a bit of an unfair comparison versus the baselines).\n", "weaknesses": "To this reviewer, it is rather hard to evaluate this work as the model and the associated losses feel very complicated. I am pretty sure I don't understand how they all play together. I think the paper could do a much better job:\n* Carefully ablating each one on a variety of datasets, extending Table 5 to consider all pairs of all losses.\n* Carefully ablating the modules and architecture choices. Table 4 seems like a good start, but it should be noted that removing the \"focus\" module for instance would presumably change the parameter count or effective footprint of the model. As a result it's hard for this reviewer to draw any conclusions there.\n\nIt would also be great to explore a little bit the role of feature extraction. The paper uses S3D feature extraction which seems important. Are there any other methods on STAR that use S3D? Would it be possible to run other baselines?\n\nNext, I think this paper could be improved if it presented a method that could be learned across video datasets. The current method, at least to the best of this reviewer's understanding, has to be trained fresh on each one. As a result it might be overfitting to some of the dataset statistics of STAR/AGQA/EgoTaskQA and as a result absorbing some amount of human task-specific supervision from those sources.\n\nFinally it woud be great to not just consider STAR/AGQA/EgoTaskQA but also other video QA datasets like MSRVTT-QA, TVQA, etc.\n\nnit: L137-138: RoBERTa doesn't use WordPiece, it uses BPE tokenization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the suggestions for a more comprehensive evaluation of our model.\n**Ablation Study On the Loss Functions**\n\nDue to space limitations, we provide a complete ablation study in the Supplementary Materials. According to the reviewer's suggestion, we try to give a more detailed analysis of the loss functions. \n\nWe carefully evaluate all pairs of all losses on all three selected datasets. The results are shown in Tab.1. \n\n**Table 1. Ablation study of the loss functions on all datasets.**\n\n| Uns/Sup | $L_{cls}^{uns}$ | $L_{iou}$    | $L_{cert}$   | $L_{cls}^{sup}$ | $L_{L1}$     | Acc on AGQA v2 | Acc on STAR (Test) | Acc on EgoTaskQA |\n| ------- | --------------- | ------------ | ------------ | --------------- | ------------ | -------------- | ------------------ | ---------------- |\n| Uns     | $\\checkmark$    | $\\checkmark$ |              |                 |              | 54.12          | 53.62              | 42.69            |\n| Uns     | $\\checkmark$    |              | $\\checkmark$ |                 |              | 52.89          | 51.35              | 41.77            |\n| Uns     |                 | $\\checkmark$ | $\\checkmark$ |                 |              | 54.27          | 53.72              | 42.85            |\n| Uns     | $\\checkmark$    | $\\checkmark$ | $\\checkmark$ |                 |              | **55.08**      | **53.86**          | **43.06**        |\n| Sup     |                 |              |              | $\\checkmark$    |              | 54.26          | 53.79              | 43.87            |\n| Sup     |                 |              |              |                 | $\\checkmark$ | 53.19          | 53.03              | 42.64            |\n| Sup     |                 |              |              | $\\checkmark$    | $\\checkmark$ | **55.08**      | **53.94**          | **44.27**        |\n\n- Firstly, the model with complete losses achieves the best results on both unsupervised and supervised settings of all datasets. It indicates that each loss plays its role in generating the event memories.\n- Secondly, under the unsupervised setting, the $L_{iou}$ shows an important role in controlling the temporal prediction of the events (2nd Row) while the losses of $L_{cls}$ and $L_{cert}$ seem not much impact when removing one of them. For supervised setting, the $L_{cls}$ shows more import role than the $L_{L1}$. \n\nThus, the conclusion is that temporal constraint is more important in an unsupervised manner. Without label information, the temporal prior is much more reliable. On the contrary, label supervision is more impact in supervised learning, since the event with explicit semantics has stronger grounding ability. Therefore, one can only contain the import losses for simplification.\n\n**Ablation Study On the Modules**\n\nTo the author's understanding, we try our best to keep the model parameters consistent. The Glance-only we used in the paper only replace the multi-level attention with the standard cross-attention. They have same parameters. See discussions with **Reviewer xLg6** about the **Effectiveness Analysis of the Focusing Chain** according to Tab.1in [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=dPUKw93v1V). Besides, as the reviewer suggested, we also verify the effectiveness of the glance module (Action Detection-Focus). See detailed analysis in the [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=t7xsOOjmQ1) with **Reviewer 16Tf**.\n\n**Table 2. Ablation study of each module on the STAR dataset.**\n\n| Model                  | Acc       |\n| ---------------------- | --------- |\n| MDETR                  | 46.26     |\n| Glance-only            | 53.12     |\n| Action Detection-Focus | 49.85     |\n| Glance-Focus           | **53.94** |\n\n**Ablation Study on Video Backbone**\n\nThe current SOTA methods evaluated on STAR datasets use different backbones. As the reviewer suggested, we try different video backbones and evaluated them on the STAR dataset.\n\n**Table 3. Ablation study of video backbone on STAR dataset.**\n\n| Model                                         | Visual Backbone | Val Acc          |\n| :-------------------------------------------- | --------------- | :--------------- |\n| [ClipBERT](https://arxiv.org/abs/2102.06183)  | ResNet50        | -                |\n| [CLIP](https://arxiv.org/abs/2103.00020)      | CLIP            | -                |\n| [RESERVE](https://arxiv.org/abs/2201.02639)   | ViT-B/32        | -                |\n| [AIO](https://arxiv.org/abs/2203.07303)       | ViT-B/32        | 47.54            |\n| [MIST-AIO](https://arxiv.org/abs/2212.09522)  | ViT-B/32        | 49.69            |\n| [MIST-CLIP](https://arxiv.org/abs/2212.09522) | CLIP            | *51.13*          |\n| **GF(uns)-S3D**                               | S3D             | **53.86(+2.73)** |\n| GF(uns)-CLIP                                  | CLIP            | 52.75(+1.62)     |\n| GF(uns)-ViT                                   | ViT-B/32        | 50.94(+1.25)     |\n\nOur GF with different visual backbones achieves SOTA performances compared to the methods using the same backbones. Therefore, the framework is effective with different backbones.\n**Generalization of the Model**\n\nCurrent VideoQA models except those large VL models all suffer from the issues of dataset transfer gap for the answer vocabularies are various. However, GF has the potential to solve various video understanding tasks, discussed in the [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=t7xsOOjmQ1). Therefore it can be extended to a VL pre-training model with stronger event reasoning ability. \n\nWhat's more, we evaluate two other VideoQA datasets in the [Global Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=LQPJMIxMFG), which shows the generalization of our model.\n\n*See limitation discussions in the [Rebuttal](https://openreview.net/forum?id=J6Niv3yrMq&noteId=ePRr6BdSrO).*"}]

Author rebuttal:
**We thank all reviewers for the insightful feedback. We are encouraged that you find our motivation interesting and reasonable. We try to address the common issues below.** 

P.S. For convenience, we directly link the paper of the related works. The best results in the tables are **bolded**, and the second-best results are in *italics*. We may cross reference to the related rebuttal.

**Selection and Analysis of the Evaluation Datasets**

We first clarify the reason for selecting the current popular evaluation benchmarks. In this paper, we mainly focus on challenging Multi-Event Video QA tasks. Motivated by **Reviewer 16Tf & yVqG**, we provide a further analysis of the current VideoQA benchmarks through comparisons on the number of events per video, average video length, and overall action classes number in Tab.1. These factors mainly come from the original dataset statistic. Numbers with * are estimated results since no annotations are provided. We comprehensively consider these factors to choose appropriate evaluation datasets for model evaluation in the main paper.

**Table 1. Datasets Analysis.**

| Datasets                                                    | # Avg. Events/ Video | Avg. Length | # Event Classes |
| :---------------------------------------------------------- | :-----------: | :---------: | --------------- |
| [MSVD-QA](https://dl.acm.org/doi/10.1145/3123266.3123427)   |      1*       |     10s     | 100*            |
| [MSRVTT-QA](https://dl.acm.org/doi/10.1145/3123266.3123427) |      1*       |     15s     | 100*            |
| [ActivityNet-QA](https://arxiv.org/abs/1906.02467)          |      1.4      |    180s     | 203             |
| [STAR](https://openreview.net/pdf?id=EfgNF5-ZAjM)           |      2.7      |     12s     | 157             |
| [AGQA](https://arxiv.org/pdf/2103.16002.pdf)                |      6.8      |     30s     | 157             |
| [EgoTaskQA](https://arxiv.org/pdf/2210.03929.pdf)           |       5       |     25s     | 793       |
| [NExT-QA](https://arxiv.org/abs/2105.08276)                 |      8.8      |     44s     | 50              |

From the table, we can find that MSVD-QA/MSRVTT-QA/ActivityNet-QA only contains about one event.  Therefore they mainly focus on single-event understanding. Note that **longer videos may not necessarily contain complex content**. Though the video lengths are relatively longer in ActivityNet-QA, they only contain one event or multiple repetitions of a single activity. The video contents in STAR/AGQA/EgoTaskQA/NExT-QA are more complex since they have multiple events happening in the video. In addition, the richness of event categories varies among different datasets. According to the statistics, there are approximately 100 common categories of events on average.

This paper mainly focuses on Multi-Event Video QA task, with the challenge of temporal and causal reasoning among multiple events. Therefore, STAR/AGQA/EgoTaskQA are selected for evaluation. Due to the lack of event-level annotation in the NExT-QA dataset, we cannot conduct experiments under supervised settings in the main paper. However, as **Reviewer VeHn** suggested, the experiments under unsupervised settings on the NExT-QA dataset have been supplemented.

**Table 2 Evaluation results on NExT-QA.**

| Methods                                       | Val Acc          | Test Acc         |
| --------------------------------------------- | ---------------- | ---------------- |
| [Just Ask](https://arxiv.org/abs/2012.00451)  | 52.3             | -                |
| [IGV](https://arxiv.org/abs/2206.02349)       | -                | 51.34            |
| [ATP](https://arxiv.org/abs/2206.01720)       | 54.3             | -                |
| [VGT](https://arxiv.org/pdf/2207.05342.pdf)   | 55.02            | 53.68            |
| [HGA+EIGV](https://arxiv.org/abs/2207.12783)  | -                | *53.7*           |
| [MIST-CLIP](https://arxiv.org/abs/2212.09522) | *57.18*          | -                |
| GF-C3D(uns)                                   | 56.33            | 54.75            |
| GF-S3D(uns)                                   | 56.44            | 55.60            |
| **GF-CLIP(uns)**                              | **58.83(+1.53)** | **57.92(+4.22)** |

We evaluate the Glance-Focus model (GF for short) on the NExT-QA dataset based on three different backbones. The GF models with the same backbone as other methods achieve SOTA results. Such results further reflect the strong multi-event reasoning ability of our method, for NExT-QA is the most complex Multi-Event VideoQA benchmark according to the analysis above.

Furthermore, as **Reviewer VeHn & yVqG** suggested, to verify our model on more VideoQA datasets, we additionally test on longer video understanding benchmarks, ActivityNet-QA. The performance shows that GF also achieves SOTA result and even surpasses the Just-Ask model pre-trained on large VideoQA data.

**Table 2 Evaluation results on ActivityNet-QA.**

| Methods                                                      | Acc            |
| :----------------------------------------------------------- | -------------- |
| [HGA](https://ojs.aaai.org/index.php/AAAI/article/view/6767) | 34.6           |
| [LocAns](https://arxiv.org/abs/2210.02081)                   | 36.1           |
| Just Ask                                                     | 36.8           |
| Just Ask(Pre-trained)                                        | *38.9*         |
| **GF(uns)**                                                  | **41.1(+2.2)** |

Meta-review: This paper has been extensively reviewed by six experts. Among those, five reviewers recommend acceptance. Overall, most of the reviewers think the method is interesting and reasonable, e.g., reviewer izc9: "The idea of glancing and focusing on multi-event VideoQA is interesting and intuitive", reviewer vehn: "a reasonable approach"; reviewer sLiX: "The proposed method simulates the reasoning process exhibited by humans, thus presenting an interesting and novel approach". However, reviewers do identify some limitations of this work, which includes the limitations in clarity of the writing, detailed analysis of experiments and comparisons with some works. Authors have addressed most of the questions raised by the reviewers. As for the concerns in terms of ablation studies (eg. backbones) and analysis (e.g., loss functions) raised by reviewer yVqG, authors have also addressed. Considering the novelties of the work and efficacy of the method, AC recommends accept, and urges the authors to incorporate the improvements and additional experiments and analysis to the final version.
Decision: Accept (poster)