SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziyi Bai $^{1,2}$ , Ruiping Wang $^{1,2}$ , Xilin Chen $^{1,2}$
<sup>1</sup>Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS),
Institute of Computing Technology, CAS, Beijing, 100190, China
$^{2}$ University of Chinese Academy of Sciences, Beijing, 100049, China
ziyi.bai@vipl.ict.ac.cn, {wangruiping, xlchen}@ict.ac.cn
Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance-Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross-attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.
For intelligent agents that can assist humans in completing daily-life tasks, complex event reasoning ability is particularly important. Multiple Event Video Question Answering (Multi-Event VideoQA) task [14, 42, 16, 43] has recently been proposed. These benchmarks require agents to reason about complex videos depicting multiple human-object interactions (i.e., events or situations). Furthermore, a wide scope of event-centric questions are involved[14], ranging from event semantic descriptions to event sequencing and dependencies to comprehensively evaluate an agent's ability to understand human behavior.
Recently, the emerged large vision-language models [1, 3, 8, 25, 47, 50] have demonstrated impressive generalization capabilities in VideoQA tasks. By leveraging pre-training on large-scale cross-modal pairs, these models can establish correlations between questions and videos to find answers effectively. Such approaches prove effective for short videos containing single or few events with one-step
Q: What did the person do to the blanket before taking the clothes?
(a) VideoQA through cross-modality correspondence modeling
(b) VideoQA through Glance-Focus
Figure 1: Comparison of different VideoQA solutions. The cross-modal common feature space is visualized. (a) Cross-modal correspondence modeling. Weak correspondence between the question with high-level event concepts and low-level lengthy video leads to a wrong answer. (b) Human-like Glance-Focus Strategy. Glancing stage: a set of episode memories are generated. Focusing stage: memory prompts are used as anchors to quickly locate the keyframes for reasoning.
questions like "What is someone doing?" However, Multi-Event VideoQA poses distinct challenge. As shown in Figure 1(a), due to the longer video duration and more events involved, it is challenging for the model to effectively find the visual clues from the lengthy video content. Additionally, the event-centric questions often include high-level event concepts while the semantics of video frames are relatively low-level, making it even more difficult to directly locate key moments related to the question.
This is very similar to the difficulties we humans encounter when addressing reading comprehension tasks, where we must quickly extract question-related context from long paragraphs. To ensure fast and effective reasoning, it is common for humans to employ a strategy of first glancing through the story, summarizing the storyline in our mind, and then using this episode memory to quickly focus on the relevant context when answering questions.
We mimic such a reasoning strategy to tackle the challenging Multi-Event VideoQA task with our proposed Glance-Focus model. It comprises two stages: glancing and focusing. The purpose of the glancing stage is to generate a set of episode memories that can efficiently summarize the multiple events occurring in the video. A naive way is to directly train an action detection model to obtain a set of actions. However, this would result in unnecessary computation, and most importantly, it may not be practical for encapsulating all possible actions in the real world using closed vocabularies.
Instead, we adopt a more flexible approach that automatically obtains a set of dynamic event memories using a Transformer Encoder-Decoder architecture [39]. For datasets with closed event vocabularies, we employ bipartite matching between the extracted memories and ground truth labels to explicitly obtain event memories. Apart from supervised event matching approach, we additionally design an unsupervised event generation way to get rid of the event-level annotations of the video. We impose individual certainty and global diversity constraints during event generation to ensure they can effectively summarize the video content.
Finally, since these generated memories are still independent events, we reorganize them into complete event memory prompts by arranging them in sequence order with temporal information remaining. As shown in Figure 1(b), since these memory prompts have high-level event semantics, they play an anchor role in the multi-modal common feature space to help quickly locate the related video content according to the question. Such as the example in Figure 1(b), we design a multi-level cross
attention mechanism to first focus on the event "putting down the blanket" before "taking the clothes" according to the question, then focus on the related video content for answer prediction.
We conduct experiments on various recently released Multi-Event VideoQA datasets, STAR [42], AGQA [14], EgoTaskQA[16] and NExT-QA[43]. The experimental results demonstrate that our proposed Glance-Focus model achieves state-of-the-art performance on all four datasets. We further verify the effectiveness of our approach via model analysis, showing the outstanding performance of key components in our method. Moreover, both quantitative and qualitative results affirm that our proposed memory prompts play a significant anchor role in multi-event reasoning by bridging the gap between questions and videos.
2.1 Video Question Answering
Early attempts[11, 49, 6, 51, 41] have made progress in improving complex spatial-temporal contextual information modeling for VideoQA tasks. They usually extract the appearance and motion features of video units (such as clips or frames) separately and then let them interact with the question to find an answer. To capture question-related spatial and temporal clues, co-memory and co-attention mechanisms are proposed by MACN[11] and HME[6] respectively. B2A[35] uses questions as a bridge to connect the appearance and motion features to find the answers. These holistic models are effective at aggregating spatial and temporal context, making them well-suited for general one-step video reasoning tasks. However, they tend to encounter challenges when confronted with more intricate compositional questions[14].
Therefore many studies[23, 15, 30, 5, 44, 45] have delved into the exploration of hierarchical modeling strategies to address the challenges of multi-grained video content representation. HCRN[23] proposes using reusable units stacked hierarchically to gradually extract frame-level to video-level features conditioned by motion and question. Graph-based methods such as L-GCN[15], HAIR[30], and VGT[45] utilize extra object detection models, like Faster R-CNN[37], to explicitly incorporate object-level features into the model. They further organize the objects into a unified graph for fine-grained reasoning. Although these hierarchical structures contain multi-grained clues, few of them explicitly model event-level structure of videos. This limitation poses a challenge in effectively handling high-level tasks, including event prediction, explanation, and counterfactual reasoning.
2.2 Video-Language Pre-trained Models
Recently, Transformers-based large video-language models (VLMs) [52, 25, 2, 8, 47, 50, 12, 1, 3, 40] have accomplished comparable or superior results on downstream tasks like VideoQA. These models harness the power of large-scale image or video-text pairs collected from the Internet to pre-train cross-modal correspondence models. They commonly employ proxy tasks like masked language modeling and masked frame modeling [25, 50, 40] or contrastive learning [47, 2] during pre-training. Some models like ActBERT[52] and BridgeFormer[12] focus on multi-grained semantics grounding by explicitly learning the alignment of objects and actions across modalities. However, applying these VLMs directly to Multi-Event VideoQA tasks is non-trivial. These models often necessitate sparse sampling of lengthy videos, leading to a notable loss of video semantics, and thus suffer from weak correspondence between questions involving high-level event concepts and intricate video content[9, 16]. To tackle this challenge, some retrieval-based cross-modal methods [48, 19, 36] have been proposed recently to efficiently extract the keyframes of the video for reasoning. Nonetheless, the use of fixed keyframes may pose difficulties when dealing with multi-grained questions. Instead, we propose to employ the dynamic and adaptable event memories for representing lengthy videos.
2.3 Multi-Event Modeling
Multi-event modeling is an essential ability for household robots to assist humans in completing complex tasks. However, explicit multi-event modeling still needs to be explored for VideoQA task. TSEA[9] proposes a heuristic method to extract event-level features of videos. Several studies targeting at multi-activity recognition and detection task[15, 34, 33] employ graph structures to model complex relationships among multiple activities. P-GCN[15] extracts activity proposals from videos and utilizes graph models to integrate context. Ego-Topo[33] organizes egocentric videos into several
Figure 2: The complex multi-event reasoning task is structured into two stages: Glancing and Focusing. At the Glancing stage, a set of unordered event memories are obtained by aggregating relevant input video frames. At the Focusing stage, these event memories are reorganized according to their sequential order. The resulting memory prompts act as bridges, allowing model progressively focus on the related video frames for answer prediction using multi-level cross-attention mechanism.
zones and learns the links between these zones for scene affordances and future action prediction. Bridge-prompt[27] leverages the continuity of the natural language to concatenate multiple discrete events by constructing a series of text prompts. Similar to the multi-activity detection task, our goal is to obtain a set of high-level key event from the video. However, directly training an action detection model requires additional event-level annotations, which may not always be available. Furthermore, it is impractical to use a closed set of actions to encompass all possible events in the real-world. By contrast, our proposed approach learns dynamic event memories from videos through either unsupervised generation or supervised set-matching methods, depending on the availability of event labels during the training process.
3 Approach: The Glance-Focus Model
We first give an overview of our model in Section 3.1. Next, we describe in detail the two stages of our model, the Glancing stage and the Focusing stage in Section 3.2 and Section 3.3 respectively. Finally, in Section 3.4 we illustrate the overall loss functions.
Given a video clip  $V$  and a question  $q$ , a VideoQA model predicts the answer  $y$ , formulated as follows:
$$
\hat {y} = \underset {y \in \mathcal {A}} {\arg \max } \mathcal {F} _ {\theta} (y | q, V, \mathcal {A}), \tag {1}
$$
where  $y$  is the predicted answer chosen from the candidate answers denoted as  $\mathcal{A}$ , i.e., answer vocabulary for open-ended questions or choices for multi-choice questions.  $\theta$  is the trainable parameters of a VideoQA model  $\mathcal{F}$ .
We propose a novel Glance-Focus VideoQA model with the Transformer Encoder-Decoder framework[39]. As shown in Figure 2, the model answers the question with two stages: 1) At the Glancing stage, a set of episode memories are queried from the input video; 2) At the focusing stage, the model reorganizes the memories into memory prompts and they will serve as anchors to localize the question-related video content by multi-level focusing.
Input representations. For the input multi-event video of length  $T$ , we use a pretrained visual backbone to extract its features  $\mathbf{X} \in \mathbb{R}^{T \times D}$  with dimension  $D$ . The video features are fed into the model both in glancing and focusing stages. For questions, following [31], we tokenize them using Byte-Pair Encoding (BPE) tokenization, and embed them with pretrained RoBERTa language
model[31] to get textual tokens  $\mathbf{w} = \{w_{1},\dots,w_{L}\}$  with question length  $L$ . Then they are projected into the same dimension feature space as video features with a projector  $\mathcal{G}(.)$  to get question embeddings  $\mathbf{L} = \mathcal{G}(\mathbf{w})\in \mathbb{R}^{L\times D}$ . The question embeddings are fed into the model only at focusing stage.
3.2 Glancing Stage
It is challenging for the VideoQA model to directly find answers from lengthy videos containing multiple complex events. To efficiently establish the corresponding relationships between the high-level event concepts of the questions and the low-level video unit features, we propose employing a quick glance strategy that effectively extracts high-level episode memories from the video to guide the subsequent reasoning.
To obtain the episode memory, a direct approach would be training an action detection model to extract a set of actions from the video. However, this may not be applicable in certain cases where video data lacks event annotations. Moreover, it is impractical to encompass all possible events in the real world with a close-set vocabulary. To address these issues, we propose a more flexible and practical solution which can automatically learn a set of dynamic memories  $\mathbf{M} = \{\mathbf{m}_1,\dots,\mathbf{m_N}\} \in \mathbb{R}^{N\times D}$  to summarize the key  $N$  events occurring in the videos. Given a group of randomly initialized memory queries, we train a Transformer encoder-decoder[39] to update them by gradually aggregating the related encoded video features  $\mathbf{X}$  through cross-attention interactions. Specifically, we design both unsupervised generation and supervised matching approaches depending on whether event-level annotations are available.
Unsupervised event memory generation. In this case, we employ the idea of information maximization[21, 29] from the unsupervised learning to generate a set of event memories. Our goal is to generate a collection of event memories. To ensure that this set comprehensively captures all the crucial events within the video, we impose two vital constraints. 1) Individual Certainty. Each individual event is encouraged to be distinctive, thereby discouraging individual frames from forming separate and independent events. 2) Global Diversity. We aim to maintain diversity within the event set, preventing all video frames from being clustered into only one single event.
Assuming that there are  $C$  types of events that will occur in the video. For individual certainty, we suppose higher classification confidence score. Namely, the ideal category distribution is similar to one-hot encoding[21]. Specifically, a classifier header  $f_{c}$  of a linear layer with softmax is used to predict the event classification logits  $\hat{\mathbf{p}}_m = f_c(\mathbf{m})$ . The individual certainty loss is denoted as
$$
\mathcal {L} _ {\text {c e r t}} \left(f _ {c}; \mathbf {M}\right) = - \mathbb {E} _ {\mathbf {m} \in \mathbf {M}} \sum_ {i = 1} ^ {C} \hat {\mathbf {p}} _ {m} ^ {i} \log \hat {\mathbf {p}} _ {m} ^ {i}. \tag {2}
$$
To ensure global diversity, we consider both the diversity of semantics and temporal spans within the event set. To promote semantic diversity, we assign category labels evenly across all memories. Namely, in contrast to the above one-hot classification logits of each event, we aim for an even distribution. The following  $\mathcal{L}_{cls}$  is minimized to constitute the semantic diversity loss
$$
\mathcal {L} _ {c l s} \left(f _ {c}; \mathbf {M}\right) = \sum_ {i = 1} ^ {C} \hat {\mathbf {p}} ^ {i} \log \hat {\mathbf {p}} ^ {i}, \tag {3}
$$
where  $\hat{\mathbf{p}} = \mathbb{E}_{\mathbf{m} \in \mathbf{M}}[f_c(\mathbf{m})]$  is the mean output embedding of the all event memories of a video sample. Next, to ensure temporal span diversity, we pursue a reduction in the temporal overlap of all events. We use a temporal header  $f_t$  of 3-layer FFN with ReLU to predict the normalized temporal center coordinate  $\hat{\delta}$  and width  $\hat{w}$  of events,  $\hat{\tau} = f_t(\mathbf{m}) = [\hat{\delta}, \hat{w}] \in [0,1]^2$ . The temporal IoU loss formulated as,
$$
\mathcal {L} _ {i o u} \left(f _ {t}; \mathbf {M}\right) = \mathbb {E} _ {\mathbf {m} \in \mathbf {M}} \sum_ {i, j = 1} ^ {N} I o U \left(\hat {\tau} _ {i}, \hat {\tau} _ {j}\right), \tag {4}
$$
where  $\mathrm{IoU}(.)$  is the normalized temporal IoU[24] given the paired temporal prediction of events. The overall global diversity loss is the weighted sum of  $\mathcal{L}_{cls}$  and  $\mathcal{L}_{iou}$
$$
\mathcal {L} _ {d i v} \left(f _ {c}; f _ {t}; \mathbf {M}\right) = \lambda_ {c l s} \mathcal {L} _ {c l s} + \lambda_ {i o u} \mathcal {L} _ {i o u}. \tag {5}
$$
Supervised event memory matching. For datasets with ground-truth event-level labels, we similarly use the set prediction via bipartite matching [4, 24] to extract event memories. Since the output of the Transformer is unordered, we do not have a one-to-one correspondence between the predicted event memories and the ground-truth events. Following [4, 24], we use the Hungarian algorithm to find the optimal bipartite matching results. Given the padding ground-truth event labels  $E = \{e_i\}_{i=1}^N$ ,  $e_i = \{c_i, \tau_i = [\delta_i, w_i]\}$ , where  $c$  and  $\tau$  are class labels and the temporal labels respectively, the matching cost  $\mathcal{C}_{match}$  is defined as:
$$
\mathcal {C} _ {\text {m a t c h}} = \sum_ {i = 1} ^ {N} \left[ - \mathbf {1} _ {c _ {i} \neq \emptyset} \lambda_ {c l s} \hat {p} _ {\pi (i)} ^ {c _ {i}} + \mathbf {1} _ {c _ {i} \neq \emptyset} \lambda_ {L 1} | | \tau_ {i} - \hat {\tau} _ {\pi (i)} | | \right], \tag {6}
$$
where  $\pi \in \Pi_N$  is the permutation of  $N$  event memories. Then the goal is to find the optimal  $\pi$  to minimize the matching cost.
Memory prompting. At the glancing stage, the output is a set of unordered event memories  $\mathbf{M} = \{\mathbf{m}_1,\dots \mathbf{m_N}\}$ . We reorganize them into  $\widetilde{\mathbf{M}}$  in the sequential order according to their temporal center coordinate  $\{\delta_i\}$ . The temporal positional embedding  $\mathbf{P}_t\in \{\Phi (i)|i\in [1,N]\}$  is added to the corresponding events to indicate their temporal positions in the video. And they will serve as the memory prompts during multi-level attention at the following focusing stage.
3.3 Focusing Stage
It is difficult to directly associate the high-level event semantics in the questions with corresponding low-level videos. Thus the event memories extracted through video glancing serve as useful anchors to prompt the model quickly locate event segments related to the questions. Video features  $\mathbf{X}$ , memory prompts  $\widetilde{\mathbf{M}} +\mathbf{P}_t$ , and question embeddings  $\mathbf{L}$  are concatenated as input. The Transformer Encoder embeds these embeddings from different sources into a joint space  $[\mathbf{X}_{enc},\mathbf{M}_{enc},\mathbf{L}_{enc}]$ .
Multi-level cross attentions. Given a set of answer queries  $\mathbf{a} \in \mathbb{R}^{N \times D}$ , the Transformer Decoder exploits multi-level cross attention to replace the original multi-head cross-attention to guide the model gradually focus on the question-related visual clues in the video.
a) Focus on memory. Given questions with high-level event semantics, we first use question embeddings as queries to interact with memory prompts to guide the model focus on the corresponding event memories  $\mathbf{M}_{\text {foc }}$ :
$$
\mathbf {Q} = \mathbf {L} _ {\text {e n c}}, \mathbf {K} = \mathbf {V} = \mathbf {M} _ {\text {e n c}} \tag {7}
$$
$$
\mathbf {M} _ {f o c} = \text {M u l t i h e a d} (\mathbf {Q}, \mathbf {K}, \mathbf {V}). \tag {8}
$$
b) Focus on frame. Then the selected event memories prompt the model to localize the corresponding video frames  $\mathbf{X}_{foc}$  as visual clues for specific reasoning:
$$
\mathbf {Q} = \mathbf {M} _ {f o c}, \mathbf {K} = \mathbf {V} = \mathbf {X} _ {e n c} \tag {9}
$$
$$
\mathbf {X} _ {f o c} = \text {M u l t i h e a d} (\mathbf {Q}, \mathbf {K}, \mathbf {V}). \tag {10}
$$
c) Predict the answer. Finally, the answer queries find the answers  $\hat{\mathbf{a}}$  through reasoning across the focused video frames:
$$
\mathbf {Q} = \mathbf {a}, \mathbf {K} = \mathbf {V} = \mathbf {X} _ {f o c} \tag {11}
$$
$$
\hat {\mathbf {a}} = \text {M u l t i h e a d} (\mathbf {Q}, \mathbf {K}, \mathbf {V}). \tag {12}
$$
Similarly to classifier header  $f_{c}$ , an answer classifier header  $f_{a}$  of a linear layer with softmax is used to predict the answer label from the last token of answer queries  $\hat{y} = f_{a}(\hat{\mathbf{a}})$ .
The first part of loss functions is the answer classification cross-entropy loss  $\mathcal{L}_{qa}$ . The overall unsupervised loss is defined as a linear combination:
$$
\mathcal {L} _ {u n s} \left(\mathcal {F} _ {\theta}; q; V; \mathcal {A}; y\right) = \mathcal {L} _ {q a} + \lambda_ {c e r t} \mathcal {L} _ {c e r t} + \mathcal {L} _ {d i v}. \tag {13}
$$
Under the situation with ground-truth event-level labels, we turn to use them for supervision by minimizing the event classification cross-entropy loss  $\mathcal{L}_{cls}$  and the temporal span  $L1$  loss  $\mathcal{L}_{L1}$  between the matching memories with the ground-truth events:
$$
\mathcal {L} _ {\sup } \left(\mathcal {F} _ {\theta}; q; V; \mathcal {A}; y; E\right) = \mathcal {L} _ {q a} + \lambda_ {c l s} \mathcal {L} _ {c l s} + \lambda_ {L 1} \mathcal {L} _ {L 1}. \tag {14}
$$
We evaluate our model on various recently proposed Multi-Event VideoQA datasets. The comparison of them is shown in Table 1. Note that we use the v2 version of the AGQA dataset, which has more balanced distributions, as the dataset creator recommended. We choose these benchmarks for a broad coverage of: Video content. Including household and general activities, etc. Video duration. The average video length ranges from 12 seconds to 44 seconds. Event complexity. All these videos contain multiple events or actions, with the average event number ranging from 2.7 to 8.8. Event complexity. The number of event categories varies from 50 to 793.