Title: Glance and Focus: Memory Prompting for Multi-Event Video Question Answering
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Question Answering; Multi-Event Reasoning; Spatial-Temporal Reasoning
Abstract: Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents’ ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance- Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross- attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.

1

Abstract

Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the GlanceFocus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level crossattention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus .

1 Introduction

For intelligent agent that can assist humans in completing daily-life tasks, complex event reasoning ability is particularly important. Multiple Event Video Question Answering (Multi-Event VideoQA) task [14, 42, 16, 43] has recently been proposed. These benchmarks require agents to reason about complex videos depicting multiple human-object interactions (i.e., events or situations). Furthermore, a wide scope of event-centric questions are involved[14], ranging from event semantic descriptions to event sequencing and dependencies to comprehensively evaluate an agent's ability to understand human behavior.

Recently, the emerged large vision-language models [1, 3, 8, 25, 47, 50] have demonstrated impressive generalization capabilities in VideoQA tasks. By leveraging pre-training on large-scale cross-modal pairs, these models can establish correlations between questions and videos to find answers effectively. Such approaches proves effective for short videos containing single or few events with one-step

video

𝒗

𝑻

𝒗

𝑻"𝟑

𝒗

𝟏

𝒗

𝒗

𝒗

𝟐

𝒗

𝑻"𝟒

video

𝒗

𝑻

𝒗

𝑻"𝟑

𝒗

𝟏

𝒗

𝒗

𝒗

𝟐

𝒗

𝑻"𝟒

...

Q

: What did the person do to the blanket before

taking the clothes?

Event 1

Event N-1

Event N

Glance: Generating Episode Memory

Focus: Reasoning with Memory Prompt

𝑣

!

𝑣

"

𝑣

#

𝑣

#$!

𝑣

#$"

𝑣

#$%

A

: Took.

(a) VideoQA through cross-modality

correspondence modeling

(b) VideoQA through Glance-Focus

Q

: What ... before

taking the clothes?

𝑣

#$&

...

𝑣

!

𝑣

"

𝑣

#

𝑣

#$!

𝑣

#$"

𝑣

#$%

𝑣

#$&

A

A

Episode

Memory

A

: Took.

taking the

clothes

putting down

the blanket

before

𝑥

"

𝑥

#

taking

the

clothes

question

before

person

taking

the

clothes

question

before

person

𝑥

"

𝑥

#

Q

: What ... before

taking the clothes?

？

…

before

before

before

G

questions like ' What is someone doing? '. However, Multi-Event VideoQA poses distinct challenge. As shown in Figure 1(a), due to the longer video duration and more events involved, it is challenging for the model to effectively find the visual clues from the lengthy video content. Additionally, the event-centric questions often include high-level event concepts while the semantics of video frames are relatively low-level, making it even more difficult to directly locate key moments related to the question.

This is very similar to the difficulties we humans encounter when addressing reading comprehension tasks, where we must quickly extract question-related context from long paragraphs. To ensure fast and effective reasoning, it is common for humans to employ a strategy of first glancing through the story, summarizing the storyline in our mind, and then using this episode memory to quickly focus on the relevant context when answering questions.

We mimic such a reasoning strategy to tackle the challenging Multi-Event VideoQA task with our proposed Glance-Focus model. It comprises two stages: glancing and focusing. The purpose of the glancing stage is to generate a set of episode memories that can efficiently summarize the multiple events occurring in the video. A naive way is to directly train an action detection model to obtain a set of actions. However, this would result in unnecessary computation, and most importantly, it may not be practical for encapsulating all possible actions in the real world using closed vocabularies.

Instead, we adopt a more flexible approach that automatically obtains a set of dynamic event memories using a Transformer Encoder-Decoder architecture [39]. For datasets with closed event vocabularies, we employ bipartite matching between the extracted memories and ground truth labels to explicitly obtain event memories. Apart from supervised event matching approach, we additionally design an unsupervised event generation way to get rid of the event-level annotations of the video. We impose individual certainty and global diversity constraints during event generation to ensure they can effectively summarize the video content.

Finally, since these generated memories are still independent events, we reorganize them into complete event memory prompts by arranging them in sequence order with temporal information remaining. As shown in Figure 1(b), since these memory prompts have high-level event semantics, they play an anchor role in the multi-modal common feature space to help quickly locate the related video content according to the question. Such as the example in Figure 1(b), we design a multi-level cross

attention mechanism to first focus on the event 'putting down the blanket' before 'taking the clothes' according to the question, then focus on the related video content for answer prediction.

We conduct experiments on various recently released Multi-Event VideoQA datasets, STAR [42], AGQA [14], EgoTaskQA[16] and NExT-QA[43]. The experimental results demonstrate that our proposed Glance-Focus model achieves state-of-the-art performance on all four datasets. We further verify the effectiveness of our approach via model analysis, showing the outstanding performance of key components in our method. Moreover, both quantitative and qualitative results affirm that our proposed memory prompts play a significant anchor role in multi-event reasoning by bridging the gap between questions and videos.

2 Related Works

2.1 Video Question Answering

Early attempts[11, 49, 6, 51, 41] have made progress in improving complex spatial-temporal contextual information modeling for VideoQA tasks. They usually extract the appearance and motion features of video units (such as clips or frames) separately and then let them interact with the question to find an answer. To capture question-related spatial and temporal clues, co-memory and co-attention mechanisms are proposed by MACN[11] and HME[6] respectively. B2A[35] uses questions as a bridge to connect the appearance and motion features to find the answers. These holistic models are effective at aggregating spatial and temporal context, making them well-suited for general one-step video reasoning tasks. However, they tend to encounter challenges when confronted with more intricate compositional questions[14].

Therefore many studies[23, 15, 30, 5, 44, 45] have delved into the exploration of hierarchical modeling strategies to address the challenges of multi-grained video content representation. HCRN[23] proposes using reusable units stacked hierarchically to gradually extract frame-level to video-level features conditioned by motion and question. Graph-based methods such as L-GCN[15], HAIR[30], and VGT[45] utilize extra object detection models, like Faster R-CNN[37], to explicitly incorporate object-level features into the model. They further organize the objects into a unified graph for fine-grained reasoning. Although these hierarchical structures contain multi-grained clues, few of them explicitly model event-level structure of videos. This limitation poses a challenge in effectively handling high-level tasks, including event prediction, explanation, and counterfactual reasoning.

2.2 Video-Language Pre-trained Models

Recently, Transformers-based large video-language models (VLMs) [52, 25, 2, 8, 47, 50, 12, 1, 3, 40] have accomplished comparable or superior results on downstream tasks like VideoQA. These models harness the power of large-scale image or video-text pairs collected from the Internet to pre-train cross-modal correspondence models. They commonly employ proxy tasks like masked language modeling and masked frame modeling [25, 50, 40] or contrastive learning [47, 2] during pre-training. Some models like ActBERT[52] and BridgeFormer[12] focus on multi-grained semantics grounding by explicitly learning the alignment of objects and actions across modalities. However, applying these VLMs directly to Multi-Event VideoQA tasks is non-trivial. These models often necessitate sparse sampling of lengthy videos, leading to a notable loss of video semantics, and thus suffer from weak correspondence between questions involving high-level event concepts and intricate video content[9, 16]. To tackle this challenge, some retrieval-based cross-modal methods [48, 19, 36] have been proposed recently to efficiently extract the keyframes of the video for reasoning. Nonetheless, the use of fixed keyframes may pose difficulties when dealing with multi-grained questions. Instead, we propose to employ the dynamic and adaptable event memories for representing lengthy videos.

2.3 Multi-Event Modeling

Multi-event modeling is an essential ability for household robots to assist humans in completing complex tasks. However, explicit multi-event modeling still needs to be explored for VideoQA task. TSEA[9] proposes a heuristic method to extract event-level features of videos. Several studies targeting at multi-activity recognition and detection task[15, 34, 33] employ graph structures to model complex relationships among multiple activities. P-GCN[15] extracts activity proposals from videos and utilizes graph models to integrate context. Ego-Topo[33] organizes egocentric videos into several

1.Glancing Stage

memory

query

...

what ...

the clothes

?

...

...

...

memory prompting

2.Focusing Stage

answer

query

A. put down

a) focus on memory

c) predict answer

Transformer Encoder

...

Transformer Decoder

Transformer Decoder

...

...

m

!

m

"

...

...

Transformer Encoder

N-2

N-1

m

#

data flow

attention

...

...

...

b) focus on frame

m

$

zones and learns the links between these zones for scene affordances and future action prediction. Bridge-prompt[27] leverages the continuity of the natural language to concatenate multiple discrete events by constructing a series of text prompts. Similar to the multi-activity detection task, our goal is to obtain a set of high-level key event from the video. However, directly training an action detection model requires additional event-level annotations, which may not always be available. Furthermore, it is impractical to use a closed set of actions to encompass all possible events in the real-world. By contrast, our proposed approach learns dynamic event memories from videos through either unsupervised generation or supervised set-matching methods, depending on the availability of event labels during the training process.

3 Approach: The Glance-Focus Model

We first give an overview of our model in Section 3.1. Next, we describe in detail the two stages of our model, the Glancing stage and the Focusing stage in Section 3.2 and Section 3.3 respectively. Finally, in Section 3.4 we illustrate the overall loss functions.

3.1 Overview

Given a video clip V and a question q , a VideoQA model predicts the answer y , formulated as follows:

where y is the predicted answer chosen from the candidate answers denoted as A , i.e., answer vocabulary for open-ended questions or choices for multi-choice questions. θ is the trainable parameters of a VideoQA model F .

We propose a novel Glance-Focus VideoQA model with the Transformer Encoder-Decoder framework[39]. As shown in Figure 2, the model answers the question with two stages: 1) At the Glancing stage, a set of episode memories are queried from the input video; 2) At the focusing stage, the model reorganizes the memories into memory prompts and they will serve as anchors to localize the question-related video content by multi-level focusing.

Input representations. For the input multi-event video of length T , we use a pretrained visual backbone to extract its features X ∈ R T × D with dimension D . The video features are fed into the model both in glancing and focusing stages. For questions, following [31], we tokenize them using Byte-Pair Encoding (BPE) tokenization, and embed them with pretrained RoBERTa language

model[31] to get textual tokens w = { w 1 , ..., w L } with question length L . Then they are projected into the same dimension feature space as video features with a projector G ( . ) to get question embeddings L = G ( w ) ∈ R L × D . The question embeddings are fed into the model only at focusing stage.

3.2 Glancing Stage

It is challenging for the VideoQA model to directly find answers from lengthy videos containing multiple complex events. To efficiently establish the corresponding relationships between the highlevel event concepts of the questions and the low-level video unit features, we propose employing a quick glance strategy that effectively extracts high-level episode memories from the video to guide the subsequent reasoning.

To obtain the episode memory, a direct approach would be training an action detection model to extract a set of actions from the video. However, this may not be applicable in certain cases where video data lacks event annotations. Moreover, it is impractical to encompass all possible events in the real world with a close-set vocabulary. To address these issues, we propose a more flexible and practical solution which can automatically learn a set of dynamic memories M = { m 1 , ..., m N } ∈ R N × D to summarize the key N events occurring in the videos. Given a group of randomly initialized memory queries, we train a Transformer encoder-decoder[39] to update them by gradually aggregating the related encoded video features X through cross-attention interactions. Specifically, we design both unsupervised generation and supervised matching approaches depending on whether event-level annotations are available.

Unsupervised event memory generation. In this case, we employ the idea of information maximization[21, 29] from the unsupervised learning to generate a set of event memories. Our goal is to generate a collection of event memories. To ensure that this set comprehensively captures all the crucial events within the video, we impose two vital constraints. 1) Individual Certainty . Each individual event is encouraged to be distinctive, thereby discouraging individual frames from forming separate and independent events. 2) Global Diversity . We aim to maintain diversity within the event set, preventing all video frames from being clustered into only one single event.

Assuming that there are C types of events that will occur in the video. For individual certainty, we suppose higher classification confidence score. Namely, the ideal category distribution is similar to one-hot encoding[21]. Specifically, a classifer header f c of a linear layer with softmax is used to predict the event classification logits ˆ p m = f c ( m ) . The individual certainty loss is denoted as

To ensure global diversity, we consider both the diversity of semantics and temporal spans within the event set. To promote semantic diversity, we assign category labels evenly across all memories. Namely, in contrast to the above one-hot classification logits of each event, we aim for an even distribution. The following L cls is minimized to constitute the semantic diversity loss

where ˆ p = E m ∈ M [ f c ( m )] is the mean output embedding of the all event memories of a video sample. Next, to ensure temporal span diversity, we pursue a reduction in the temporal overlap of all events. We use a temporal header f t of 3-layer FFN with ReLU to predict the normalized temporal center coordinate ˆ δ and width ˆ w of events, ˆ τ = f t ( m ) = [ ˆ δ, ˆ w ] ∈ [0 , 1] 2 . The temporal IoU loss formulated as,

where IoU(.) is the normalized temporal IoU[24] given the paired temporal prediction of events. The overall global diversity loss is the weighted sum of L cls and L iou

Supervised event memory matching. For datasets with ground-truth event-level labels, we similarly use the set prediction via bipartite matching [4, 24] to extract event memories. Since the output of the Transformer is unordered, we do not have a one-to-one correspondence between the predicted event memories and the ground-truth events. Following [4, 24], we use the Hungarian algorithm to find the optimal bipartite matching results. Given the padding ground-truth event labels E = { e i } N i =1 , e i = { c i , τ i = [ δ i , w i ] } , where c and τ are class labels and the temporal labels respectively, the matching cost C match is defined as:

̸

̸

where π ∈ Π N is the permutation of N event memories. Then the goal is to find the optimal π to minimize the matching cost.

Memory prompting. At the glancing stage, the output is a set of unordered event memories M = { m 1 , ... m N } . We reorganize them into ˜ M in the sequential order according to their temporal center coordinate { δ i } . The temporal positional embedding P t ∈ { Φ( i ) | i ∈ [1 , N ] } is added to the corresponding events to indicate their temporal positions in the video. And they will serve as the memory prompts during multi-level attention at the following focusing stage.

3.3 Focusing Stage

It is difficult to directly associate the high-level event semantics in the questions with corresponding low-level videos. Thus the event memories extracted through video glancing serve as useful anchors to prompt the model quickly locate event segments related to the questions. Video features X , memory prompts ˜ M + P t , and question embeddings L are concatenated as input. The Transformer Encoder embeds these embeddings from different sources into a joint space [ X enc , M enc , L enc ] .

Multi-level cross attentions. Given a set of answer queries a ∈ R N × D , the Transformer Decoder exploits multi-level cross attention to replace the original multi-head cross-attention to guide the model gradually focus on the question-related visual clues in the video.

a) Focus on memory. Given questions with high-level event semantics, we first use question embeddings as queries to interact with memory prompts to guide the model focus on the corresponding event memories M foc :

b) Focus on frame. Then the selected event memories prompt the model to localize the corresponding video frames X foc as visual clues for specific reasoning:

c) Predict the answer. Finally, the answer queries find the answers ˆ a through reasoning across the focused video frames:

Similarly to classifier header f c , an answer classifier header f a of a linear layer with softmax is used to predict the answer label from the last token of answer queries ˆ y = f a ( ˆ a ) .

3.4 Loss Functions

The first part of loss functions is the answer classification cross-entropy loss L qa . The overall unsupervised loss is defined as a linear combination:

Under the situation with ground-truth event-level labels, we turn to use them for supervision by minimizing the event classification cross-entropy loss L cls and the temporal span L 1 loss L L 1 between the matching memories with the ground-truth events:

4 Evaluation

4.1 Datasets

We evaluate our model on various recently proposed Multi-Event VideoQA datasets. The comparison of them is shown in Table 1. Note that we use the v2 version of the AGQA dataset, which has more balanced distributions, as the dataset creator recommended. We choose these benchmarks for a broad coverage of: Video content . Including household and general activities, etc . Video duration . The average video length ranges from 12 seconds to 44 seconds. Event complexity . All these videos contain multiple events or actions, with the average event number ranging from 2.7 to 8.8. Event complexity . The number of event categories varies from 50 to 793.

4.2 Implementation Details

For each benchmark, we follow standard protocols outlined by prior works for data processing, metrics, and settings to ensure fair comparisons. For video feature extraction, we use frozen S3D model [46] pre-trained on HowTo100M[32] on STAR dataset and C3D[38] feature on EgoTaskQA dataset following[47, 16]. We also try Faster-RCNN model[37] pretrained on the Visual Genome[22] and CLIP(ViT-B/32) for more challenging AGQA and NExT-QA datasets. We employ a standard 2-layer, 8-head Transformer Encoder-Decoder with hidden size D of 512 as the backbone for our Glance-Focus model. According to the statistics of common event categories, the query number N is set to 10 since the maximum average number of events per video in the current dataset is approximately 8.8. And the event class number C is set to the ground truth overall class number in the dataset or 100 if it is unknown by default. For training details, we use dropout of 0.1, and initialize model weights using Xavier init[13]. Adam optimizer[20] is used with a learning rate of 5e-6 to optimize model parameters. All experiments are conducted on an NVIDIA GeForce RTX 3090Ti GPU. More details can be found in the supplementary material.

4.3 Comparison with State-of-the-arts

We conduct a comprehensive comparison with current state-of-the-arts (SOTA) VideoQA methods, including traditional spatial-temporal models and large vision language models. We compare our Glance-Focus model (shorted as GF model in the following) with them on all four datasets. The results are shown in Table 2, 3, 4 and 5 respectively. Overall, our proposed GF model achieves SOTA results on all benchmarks, and obtains significant advantages in both the STAR and the EgoTaskQA dataset. We will analyze different datasets one by one as follows. Note that in Table 2-5 the red numbers highlight the significant gains compared to prior SOTA results.

For STAR dataset, our proposed model obtains excellent reasoning performance shown in Table 2. When facing all four different types of questions, GF significantly surpasses the previous SOTAs, especially on the Sequence category. Such results reflect that our model has strong temporal contextual modeling ability. What's more, note that our model trained without ground-truth event labels, i.e. , GF(uns), has achieved comparable results with the one trained under supervision, i.e. , GF(sup). It indicates that our proposed unsupervised memory generation in the Glance stage can effectively extract key events with strong discriminability.

We also evaluate our model in the normal split of the EgoTaskQA dataset. From Table 3, we can find that our model outperforms SOTA methods on almost all question types. Our unsupervised GF model has strong performance especially on the intent and explanatory question types. These questions evaluate the understanding of the humans daily activities, which are all necessary for building the household robots. And the model trained under supervision achieves strong performance especially on object and predictive question types, which indicates that by explicitly modeling the events happening in the video, the model can obtain stronger event semantic understanding ability, namely, to recognize the interacted objects and to predict the next action.

As for AGQA dataset, from Table 4, our model also achieves the SOTA result with object-level video features, i.e. , GF(sup) with Faster R-CNN[37] backbone. And the one trained with S3D features also gains comparable results. We show several representative reasoning types of the dataset. Our model obtains better results on some challenging question types, such as obj-rel composition questions and temporal superlative questions like ' the longest... '. Overall, our model shows more clear advantage over SOTA on the challenging Open questions, compared to the relatively simple Binary questions for which the model could gain a 50% accuracy rate by blind guessing. The complete breakdown results on all kinds of question types and a more thorough analysis can be found in the Supplementary Material.

We also evaluate our model on the challenging NExT-QA dataset based on different video backbones. Note that the NExT-QA dataset does not provide event annotations. Therefore, we exploit the proposed unsupervised memory generation in the Glance stage. As shown in Table 5, without event-level supervision or applying any extra action detection model, the unsupervised GF models with the same backbone as other methods still achieve SOTA performances. Such results further justify the strong multi-event reasoning ability of our method, considering that NExT-QA is currently the most complex Multi-Event VideoQA benchmark with longer video duration (44s) and richer events (8.8 actions per video).

What's more, we also report the breakdown results by the different question types. According to the results, GF has strong event reasoning ability especially on Descriptive (Acc@D) questions, improving the accuracy by 3.61%, and also on event relationship reasoning, achieving 2.31% accuracy gain on Causal (Acc@C) questions. As recommended in ATP[3], we additionally report results on its proposed ATP-hard subset of the NExT-QA validation set. This subset filters out those 'easy' questions that can be answered with a single frame. The results in Table 5 show that our GF models significantly surpass HGA[17] models on this subset by improving 5.22% accuracy overall. The above results indicate that compared to HGA and other methods that directly model the correspondence between question and long video through co-attention, our method using event memory as an intermediate bridge to establish such correlation for reasoning is more efficient.

4.4 Model Analysis

In this section, we evaluate the effectiveness of our proposed modules by comparing different variants of the GF models and various baselines. We first introduce the baselines and GF variants as follows. Baselines : 1) MDETR [18] which utilizes same 2-layer Transformer Encoder-Decorder architecture. 2) An Oracle model with ground-truth event labels concatenated to the questions as textual input as an upper-bound model. GF Variants : 1) Glance-only . Model which uses standard cross-attention to replace the multi-level attention strategy. 2) Detect-Focus . Apply a frozen action detection model, such as SlowFast[7], to replace the Glance stage of the model. We try to keep the model parameters consistent.

Effect of memory prompting. Firstly, the performance of the Oracle model significantly improves compared to the baseline model. It convinced us that event memories do play an important prompting role in the Multi-Event VideoQA task. Besides, it can be found that with event memory generation at the glancing stage, the Glance-only model outperforms the MDETR baseline and the model using a frozen action detection (Detect-Focus) with a large gap on both datasets. Such results reflect the effectiveness of our proposed memory-prompting strategy. Compared to additionally fine-tuning an action detection model, our proposed Glance strategy can adaptively generate key event/action features of the long videos without any annotations.

What's more, we evaluate the effectiveness of different loss functions that are designed during the event memory extraction in Table 7. We carefully evaluate all pairs of losses on STAR, EgoTaskQA and AGQA datasets. Firstly, the model with complete losses achieves the best results on both unsupervised and supervised settings of all datasets. It indicates that each loss plays its role in generating the event memories. Secondly, under the unsupervised setting, the L iou shows an

important role in controlling the temporal prediction of the events. For supervised settings, the L cls shows a more important role for the event semantic supervision helps cross-modal grounding. Please see more discussions on the loss functions in the Supplementary Material.

Effect of multi-level attention. From Table 6, compared to the Glance-only model using standard cross-attention, the GlanceFocus model with our designed multi-level attention shows stronger reasoning ability on both STAR and EgoTaskQA datasets. Taking an example from STAR dataset, we further visualize the attention weights in different levels in Figure 3, the darker the color, the greater the weight. The attention map in the first row shows the correspondence of the question embeddings with the generated memories. In this level, the model focuses on the most related event ' Throwing a towel '. The attention map in the second row is the correspondence between the focused event and the

video embeddings. At this level, the model correctly attends to the most related keyframes for answer prediction.

Someone is eating something

(0.2023, 0.4047)

Putting some food somewhere

(0.2093, 0.4186)

Washing something with a towel

(0.3744, 0.2558)

Throwing a towel/s

somewhere

(0.4279, 0.2233)

Holding a dish

(0.2413, 0.3042)

Putting some food somewhere

(0.2594, 0.4277)

Throwing a towel/s somewhere

(0.4612, 0.2537)

Washing something with a towel

(0.3801, 0.2693)

Watching/Looking outside of a window

(0.7395, 0.6605)

Holding a phone/camera

(0.7805, 0.6305)

GT:

[event;

(center,

width)]

Pred:

Which

object

did

the

person

throw

after

they

put

the

down

food?

Holding a dish

Putting some food somewhere

Washing something with a towel

Throwing a towel/s somewhere

Holding a phone/camera

Video:

Throwing

a towel/s

somewhere

𝑡

frame 1

frame

𝑇

(𝑀

!"#

)

Question: Which object did the person throw after they put down the food?

Predicted Answer: The towel.

5 Conclusion

Our proposed Glance-Focus model provides an innovative approach to the challenging Multi-Event VideoQA task. With a set of event memories as prompts, the model can efficiently establish the correspondence between high-level questions and low-level video frames. Our experiments has shown that event memories play a crucial role in locating question-related visual clues for reasoning. The proposed Glance-Focus strategy can obtain a complete and compact representation of videos without additional annotations, which can be extended to more video comprehension tasks, such as video captioning/summarization. We hope this study can inspire more future research in the field of Multi-Event Video Understanding.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 2022.

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1728-1738, 2021.

Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the 'video' in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2917-2927, 2022.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision , pages 213-229, 2020.

Long Hoang Dang, Thao Minh Le, Vuong Le, and Truyen Tran. Hierarchical object-oriented spatiotemporal reasoning for video question answering. International Joint Conference on Artificial Intelligence , 2021.

Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1999-2007, 2019.

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6202-6211, 2019.

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: Endto-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 , 2021.

Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: a video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1675-1685, 2021.

Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. arXiv preprint arXiv:2212.09522 , 2022.

Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6576-6585, 2018.

Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridgeformer: Bridging video-text retrieval with multiple choice questions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11287-11297, 2021.

Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. Location-aware graph convolutional networks for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11021-11028, 2020.

Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems , 2022.

Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11109-11116, 2020.

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021.

Sungdong Kim, Jin-Hwa Kim, Jiyoung Lee, and Minjoon Seo. Semi-parametric video-grounded text generation. arXiv preprint arXiv:2301.11507 , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2015.

Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized information maximization. Advances in Neural Information Processing Systems , 23, 2010.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision , 123:32-73, 2017.

Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9972-9981, 2020.

Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34:11846-11858, 2021.

Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7331-7341, 2021.

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Bridge-prompt: Towards ordinal action understanding in instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19880-19889, 2022.

Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2928-2937, 2022.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning , pages 6028-6039, 2020.

Fei Liu, Jing Liu, Weining Wang, and Hanqing Lu. Hair: Hierarchical visual-semantic relational reasoning for video question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1698-1707, 2021.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2630-2640, 2019.

Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 163-172, 2020.

Megha Nawhal and Greg Mori. Activity graph transformer for temporal action localization. arXiv preprint arXiv:2101.08540 , 2021.

Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Bridge to answer: Structure-aware graph interaction network for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15526-15535, 2021.

Tianwen Qian, Ran Cui, Jingjing Chen, Pai Peng, Xiaowei Guo, and Yu-Gang Jiang. Locate before answering: Answer guided question localization for video question answering. IEEE Transactions on Multimedia , 2023.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems , pages 91-99, 2015.

Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4489-4497, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Jianyu Wang, Bing-Kun Bao, and Changsheng Xu. Dualvgr: A dual-visual graph reasoning unit for video question answering. IEEE Transactions on Multimedia , pages 3369-3380, 2021.

Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Advances in Neural Information Processing Systems , 2021.

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9777-9786, 2021.

Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video as conditional graph hierarchy for multi-granular question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.

Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. Video graph transformer for video question answering. In Proceedings of the European Conference on Computer Vision , pages 39-58, 2022.

Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European

Conference on Computer Vision , pages 305-321, 2018.

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1686-1697, 2021.

Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988 , 2023.

Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6281-6290, 2019.

Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Multimodal neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhenxin Xiao, Xiaohui Yan, Jun Yu, Deng Cai, and Fei Wu. Long-form video question answering via dynamic hierarchical reinforced networks. IEEE Transactions on Image Processing , pages 5939-5952, 2019.

Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8746-8755, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a model for video question answering called \"Glance-Focus\". The idea is to first \"Glance\" at a video to obtain a set of N dynamic memories, each represented as vectors of dimension D. \n\nThe memories can either be learned in an unsupervised way or a supervised way. In the unsupervised case, the paper uses a collection of losses encouraging the video frames be assigned to different memory cells. In the supervised case, where there are ground-truth event labels that are localized in time, the paper uses bipartite matching to match the dynamic memories with the ground truth event labels, and that forms the basis of the loss.\n\nNext, the paper uses an encoder-decoder model. The encoder encodes the video features, memory cells, and question embeddings all together. Then, the decoder starts with a representation of the questio, and cross-attends to the memory cells, then to the video frames, and then to the answer.\n\nThe model is initialized from S3D pretrained on HowTo100M, and learned on STAR, AGQA, and EgoTaskQA. The model seems to do well on all the datasets.\n\n--\nupdate - I still agree with my concerns post-rebuttal and would like to keep my score.\n\nSTRENGTHS:\n* The core idea that this paper seems to be tackling is an important one. Videos are long and redundant, and dealing with this space better is an important area of research.\n* The model seems to perform quite well on the STAR test set - (though, it should be noted, it was trained on the raw videos from that dataset, so it might be a bit of an unfair comparison versus the baselines).\n\nWEAKNESSES:\nTo this reviewer, it is rather hard to evaluate this work as the model and the associated losses feel very complicated. I am pretty sure I don't understand how they all play together. I think the paper could do a much better job:\n* Carefully ablating each one on a variety of datasets, extending Table 5 to consider all pairs of all losses.\n* Carefully ablating the modules and architecture choices. Table 4 seems like a good start, but it should be noted that removing the \"focus\" module for instance would presumably change the parameter count or effective footprint of the model. As a result it's hard for this reviewer to draw any conclusions there.\n\nIt would also be great to explore a little bit the role of feature extraction. The paper uses S3D feature extraction which seems important. Are there any other methods on STAR that use S3D? Would it be possible to run other baselines?\n\nNext, I think this paper could be improved if it presented a method that could be learned across video datasets. The current method, at least to the best of this reviewer's understanding, has to be trained fresh on each one. As a result it might be overfitting to some of the dataset statistics of STAR/AGQA/EgoTaskQA and as a result absorbing some amount of human task-specific supervision from those sources.\n\nFinally it woud be great to not just consider STAR/AGQA/EgoTaskQA but also other video QA datasets like MSRVTT-QA, TVQA, etc.\n\nnit: L137-138: RoBERTa doesn't use WordPiece, it uses BPE tokenization\n\nQUESTIONS:\nWhat do you see the role of each of the losses? Are all important, and could any be removed? Are there any architecture ablations that could be done while keeping the model size the same?\n\nLIMITATIONS:\nnope, would be great to add such a section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach for multi-event video question answering (VideoQA) that mimics the human reasoning strategy of glancing and focusing. The proposed method consists of two stages: glancing and focusing. In the glancing stage, the model generates a set of episode memories that summarize the key events in the video and supports both unsupervised and supervised methods, depending on the availability of event-level annotations. In the focusing stage, it uses the episode memories as memory prompts to guide the multi-level cross-attention between questions and videos, and predicts the answer based on the focused video frames. The paper evaluates the model on three challenging multi-event VideoQA benchmarks: STAR, AGQA, and EgoTaskQA. The experimental results show that it achieves state-of-the-art performance on all datasets, and outperforms current large vision-language models in various complex reasoning tasks. The paper also conducts ablation studies and visualizations to demonstrate the effectiveness of the proposed modules and memory prompts.\n\nSTRENGTHS:\n1. The proposed method simulates the reasoning process exhibited by humans, namely an initial overview followed by a concentrated examination, thus presenting an interesting and novel approach.\n2. The implementation is flexible and practical, enabling the simultaneous processing of annotated and unannotated videos without dependence on predetermined action vocabularies or event-level annotations.\n3. This approach represents the pioneering effort to incorporate information maximization as a guiding principle for the training of unlabeled data within the realm of VideoQA.\n4. It designs a multi-level cross-attention mechanism that leverages the episode memories as prompts to guide the model to focus on the relevant video frames for answer prediction.\n5. The performance is quite impressive, as it even surpasses current large vision-language models in various scenarios.\n\nWEAKNESSES:\n1. The writing has a large room to be improved.\n2. In L166, the proposed method assumes there are C events in a video. However, it remains unclear whether C keeps consistent across all videos or dynamically varies for every single one. If C is predetermined, it would be valuable to conduct an ablative study to assess the impact of different choices for C on the overall performance of the method.\n3. For unsupervised event generation, the authors impose a strong bias that the category labels should be evenly distributed across all memories. Further elaboration is essential to explain its reasonability. As from a subjective standpoint, only a limited subset of categories can manifest in a video given the contextual constraints (e.g., playing football on the ocean is impossible).\n\nQUESTIONS:\nplease refer to weaknesses.\n\nLIMITATIONS:\nThe limitations have been well discussed in supplementary materials. The potential negative societal impact is recommended to be addressed as well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage method of constructing memories of periodical summarization of the video, then using memory prompts to local key frames for reasoning.\n\nSTRENGTHS:\n1. The SOTA results on three Ego QA datasets.\n2. The hierarchical construction of video abstraction might be a new way of tackling lengthy videos.\n3. The IoU based un-unsupervised learning could be extended to other senarios.\n\nWEAKNESSES:\nThe writing needs improvement. Sentences are generally long and hard to understand. \nE.g., typo in abstract \"abtain\".  Long sentence such as line 13-17. \n\n\nThe idea of one-to-one matching of discrete events with memories seem to be outdated. Why not use soft-attention to assign soft weights of each event to the memory. This seems to be a well-studied technique.\n\n\nThe dependency in a)-c) focus is not justified. Would this chain of dependency bring in heavy loss if M_enc is not accurate. The un-supervised construction of heuristic M seems to be not a suitable prompt. What would (9) be Q=M_foc + L_enc? Any ablation about this.\n\nThe results seem not convincing in terms of the proposed focus chain. Would this accuracy improvement come from the additional parameters?\n\nSeems to be not a fair comparison with non-memory methods. Would methods like MIST. What would a baseline that does not construct M_enc explicitly but implicitly construct a M_foc do. Does the additional steps of constraining memory M from Eq2-5 harm the performance?\n\nQUESTIONS:\nThe results seem not convincing in terms of the proposed focus chain. Would this accuracy improvement come from the additional parameters?\n\nHow is the positional embedding \\phi formulated in Memory prompting? If it's from bi-partite assignment, is it trainable ? \n\nNot clear how IoU is in (4). Can you provide more details.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to tackle the problem of multi-event VideoQA that involves multiple human-object interaction events. Inspired by the human's reasoning ability, this paper proposes the Glance-Focus model that can reason from coarse to fine. At the glancing stage, the Glance-Focus model can generate some dynamic key event memories to summarize the key events in the video. At the focusing stage, the Glance-Focus model can reason based on the most relevant moment through the cross-attention mechanism. The contributions of this paper are summarized as following:\n(1) This paper proposes a glance-and-focus mechanism to reason multi-event VideoQA in a human-like manner.\n(2) The Glance-Focus model achieves the state-of-the-art performance on some multi-event VideoQA dataset.\n\nSTRENGTHS:\n(1) The idea of glancing and focusing on multi-event VideoQA is interesting and intuitive. This human-like design brings great potential for solving such multi-event VideoQA problem. The glancing can be seen as a kind of information compression to anchor key events, which may guide the focusing stage to conduct the fine-grained reasoning.\n\n(2) The motivation of unsupervised event generation at the glancing stage is clear. Because of the open-vocabulary world, it is hard to annotate all the events in the long-time video. To tackle this problem, this paper proposes to enforce the unsupervised constraints to generate meaningful key events.\n\n(3) To generate useful event memory in an unsupervised manner, this paper proposes to enforce the individual discrimination and the global diversity, which can guarantee to learn more meaningful key events.\n\n(4) This paper innovatively proposes to use event memories as anchors to prompt the model to locate the most relevant segments corresponding to the questions. Besides, the multi-level cross-attention mechanism can promote the event memory to focus on the question-related visual clues in a fine-grained manner.\n\n(5) Extensive experiments and analyses show the effectiveness and reasonability of the method. Specifically, the visualization of Figure 4 clearly shows the relationship between the event memory generation and the fine-grained focusing.\n\nWEAKNESSES:\n(1) At the glancing stage, this paper utilizes the unsupervised learning solution to generate meaningful event memories. However, this paper lacks of the analyses about the inherent relations between the unsupervised learning (such as instance discriminative learning) and the key event memory generation.\n\n(2) The formulation of the temporal IoU loss (Eq.(4)) is not very clear and it should be explained more detailedly.\n\n(3) In experiments, some unsupervised results even outperform the supervised manner but necessary analyses about this phenomenon are lacking.\n\nQUESTIONS:\n(1) The Glance-Focus model can not only tackle the problem of multi-event VideoQA but also tackle other kinds of reasoning problems. What about its scalability? The authors should provide more analyses.\n\n(2) In the unsupervised manner at the glancing stage, how to ensure the sequential order of events?\n\nLIMITATIONS:\nYes, the authors have discussed the limitations in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a model called Glance-Focus for Video Question Answering (VideoQA) tasks, which involve understanding human behaviors in videos. While vision language models have shown success in multi-modal tasks, reasoning over long videos with multiple human-object interaction events remains challenging. Humans can tackle this task effectively by using memory to quickly locate key moments for reasoning. To mimic this reasoning strategy, the Glance-Focus model is proposed.\n\nAt the glancing stage, an Encoder-Decoder is trained to generate dynamic key event memories by aggregating relevant video frames. This is achieved through unsupervised memory generation using individual discriminating and global diversity loss, as well as supervised bipartite matching losses. At the focusing stage, the event memories enable the model to focus on the key event memory and then on the most relevant moment for specific reasoning using multi-level cross-attention. \n\nThe model is evaluated on three representative VideoQA benchmarks (STAR, EgoTaskQA, and AGQA) and achieves state-of-the-art results, surpassing current large models in challenging tasks. The Glance-Focus model addresses the challenge of situation reasoning in VideoQA by generating dynamic event memories and using them to focus on key moments in videos.\n\nSTRENGTHS:\nThe paper's organization is good, it clearly explains the basic ideas. \n\nThe paper proposed to involve memory to cross the gap between the concepts and video content in the QA transformers.\n\nThe related work is organized well and will be better to introduce the recent benchmarks together with the VideoQA.\n\nThe implemented experiments are sufficient and the example visualization is helpful.\n\nWEAKNESSES:\nFirstly, in Table 1, AIO / Temp[ATP] / MIST results appear to be directly cited from [1] but seem to be based on the STAR validation part rather than the test part. Only RESERVE is reported on the STAR test. The dataset partition used for evaluation must be clarified for a meaningful comparison.\n\nIn Section 4.1, please specify the average video length of EgoTaskQA. If available, additional statistics, such as the average number of events/action state changes per dataset, would be beneficial since the proposed method detects state changes in the 'glance' stage. This data will aid in understanding how the method handles scenes with varying event complexity.\n\nFurthermore, the paper lacks qualitative evaluation metrics for action/event/moment detection, such as IoU/mAP. These are essential for analyzing accumulated errors.\n\nThe paper asserts the impracticality of using an action detector/event detector due to the close-set results it produces. However, a qualitative comparison with the proposed method using event/action detectors + rule-based filtering (keywords matching), e.g. Moment-DETR [2] and TubeDETR [3], is absent.\n\nLastly, although the paper discusses multi-event detection, it still chooses multiple keyframes/visual clues in the focusing stages, which has been explored in previous works [4] and [5, 6]. A more comprehensive discussion comparing these related works would be appreciated.\n\nSome typos: Line 233 HowTo100m → HowTo100M.\n\n\n[1] D. Gao et al, MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering\n[2] J. Lei et al, Detecting moments and highlights in videos via natural language queries\n[3] A, Yang et al, TubeDETR: Spatio-Temporal Video Grounding With Transformers\n[4] S. Yu et al, Self-Chained Image-Language Model for Video Localization and Question Answering\n[5] S. Kim, Semi-parametric video-grounded text generation\n[6] T. Qian et al, Locate before answering: Answer guided question localization for video question answering\n\nQUESTIONS:\nRefer to the above sections.\n\nLIMITATIONS:\nThe codes should be open-source but it seems the paper doesn't mention", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of question-answering in videos which have multiple events. The authors draw inspiration from the human capability to create event memories as a way to anchor the reasoning process over multiple events, and propose the Glance-Focus model, which aggregates a set of event memories (during a first \"glancing\" stage) and then proceeds to further process these memories (\"focus\" stage) to produce the final answer to the query. The authors benchmark their model on several standard multi-event benchmarks, such as STAR, AGQA (the balanced v2 version), and EgoTaskQA to characterize the effectiveness of their approach.\n\nSTRENGTHS:\n`+` Question-answering in videos is an important task and challenging domain for multimodal models. Further, the goal of temporal reasoning is an important step for the development of AI agents in general.\n\n`+` The general direction of reasoning on top of generated/matched event memories draws inspiration from human event memory/reasoning, and seems like a reasonable approach.\n\n`+` The authors evaluation their technique on several benchmarks (representing different video distributions) and show improvements; I also appreciated that the authors looked at the v2 version of AGQA, for better characterization of the model capabilities.\n\nWEAKNESSES:\n`-` The paper emphasizes \"long video\" understanding, but the length of the videos in the datasets examined are generally quite short (12-30 seconds) relative to many other video QA settings (e.g., ActivityNet-QA is ~2 mins). I understand this may be an inherited term from the prior work, MIST [10], which also uses similar terminology. However, that prior work did at least evaluate on NExT-QA, which is ~50% longer on average than the longest video dataset examined in this work here (AGQA).\n\n`-` Additionally, while the results on the short benchmark (STAR) are significant, the results become significantly more mixed when compared with closely related work (e.g., [10]) on longer benchmarks like AGQAv2 (+0.69 overall, and some specific categories like \"sequencing\", which seems like it should be better with event reasoning, are much worse than prior work). This also brings to question the efficacy of the proposed approach on longer video settings.\n\n`-` Further, the mixed results are even more striking due to the fact that this work leverages stronger low-level event supervision (for the bipartite event memory matching) than prior work (e.g., MIST) does. It is unclear how, without this level of supervision, the gains of the proposed technique, relative to prior work, are impacted (or the limitations of this approach as the videos become longer).\n\nQUESTIONS:\nOverall, the rating for this paper pre-rebuttal is borderline. If the authors can address and clarify the points/questions raised in the weaknesses above, that would be helpful for updating the final rating and better grounding the discussion phase with the other reviewers.\n\nLIMITATIONS:\nThe authors don't seem to provide a detailed discussion of limitations in the work, though the societal impacts does seem to be similar to related work in this space. It would be helpful and clarifying if there was an better indicator of how aspects of the proposed technique, e.g. the supervised bipartite matching, scale with longer videos, and how such limitations may be better addressed by future work.\n\n---\n\n**Post-rebuttal update:** The authors largely address some of the key concerns that were brought up in the original review, in particular, the additional analysis on NExT-QA significantly improves the evaluation (the results on ActivityNet are promising, but not the *primary* focus of the updated review given the context and discussion below). While there are some still areas that can still be further improved (as detailed and described in the rebuttal discussion below), I am increasing my rating towards leaning acceptance since I believe that these are areas that can be further addressed by future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}