Semantic-aware Dynamic Retrospective-Prospective Reasoning for Event-level Video Question Answering
Year: 2023

Abstract:
Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to obtain the visual information needed to provide optimal answers. However, despite significant progress in model performance, few studies have focused on using the explicit semantic connections between the question and visual information especially at the event level. There is need for using such semantic connections to facilitate complex reasoning across video frames. Therefore, we propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering. Specifically, we explicitly use the Semantic Role Labeling (SRL) structure of the question in the dynamic reasoning process where we decide to move to the next frame based on which part of the SRL structure (agent, verb, patient, etc.) of the question is being focused on. We conduct experiments on a benchmark EVQA dataset TrafficQA. Results show that our proposed approach achieves superior performance compared to previous state-of-the-art models. Our code is publicly available at https://github.com/ lyuchenyang/Semantic-aware-VideoQA.

heading: 1 Introduction
text: This paper focuses on one specific variant of Video Question Answering (VQA) (Xu et al., 2016; Yu et al., 2018; Zhong et al., 2022), namely Eventlevel VQA (EVQA) (Xu et al., 2021). In general, the objective of the VQA task is to provide an answer to a visual-related question according to the content of an accompanying video. Despite significant recent progress in VQA, EVQA still remains one of the most challenging VQAbased tasks since it requires complex reasoning over the events across video frames (Sadhu et al., 2021; Zhong et al., 2022; Liu et al., 2022). To
∗*corresponding author
tackle the challenges in EVQA, a number of approaches have been proposed (Xu et al., 2021). Luo et al. (2022) propose a temporal-aware bidirectional attention mechanism for improving event reasoning in videos, while Zhang et al. (2022) propose a novel model named Energy-based Refinedattention Mechanism (ERM), which obtains better performance compared to previous approaches with a smaller model size. Liu et al. (2022), on the other hand, incorporate visual-linguistic causal dependencies based on Graph Convolutional Networks (Kipf and Welling, 2017) for enhancing cross-modal event reasoning for EVQA.
Despite recent advances, conventional EVQA approaches generally fail to take into account the explicit semantic connection between questions and the corresponding visual information at the event level. Therefore, we propose a new approach that takes advantage of such semantic connections, using the Semantic Role Labeling (SRL) (Màrquez et al., 2008; Palmer et al., 2010; He et al., 2017) structure of questions. The model uses SRL information to learn an explicit semantic connection between the text-based questions and visual information in videos. Additionally, we carry out a multi-step reasoning mechanism over video frames to avoid adapting to spurious correlation and shortcuts by explicitly learning the reasoning process itself (Yi et al., 2018; Zhang et al., 2021; Picco et al., 2021; Hamilton et al., 2022; Zhu, 2022).
Specifically, in each reasoning step, the model should explicitly decide which frame should be focused on by predicting the reasoning direction (retrospective or prospective). In terms of the question, in each reasoning step, we focus on one or more specific SRL arguments with high attention weights, and model its connection with the visual information (i.e., video frames) contained within the corresponding video. For example, for a question such as [ARG1: How many cars] were [Verb: involved] [ARG2: in the accident?], the model con-
50
centrates on the ARG2 when locating the accident, before determining how many cars were involved in the accident (ARG1). In a specific reasoning step, t, we inject the relevant visual information based on the semantic connection between the question and video frames by updating a hidden vector. This vector is ultimately expected to contain the necessary information for predicting the correct answer. In the reasoning process, we employ a coverage mechanism (Tu et al., 2016) to improve the coverage of the SRL arguments of question. Namely, instead of simply focusing on a small number of specific arguments, the model is capable of including a large range of arguments.
To investigate the effectiveness of the proposed approach, we conduct experiments on a benchmark EVQA dataset: TrafficQA. Results reveal the model to achieve performance superior to that of existing baselines for a range of reasoning types (e.g., counterfactual, prospective).

heading: 2 Methodology
text: An overview of our approach is shown in Figure 1. Suppose the input of our model consists of a video V composed of n image frames sampled from it: V = {f0, f1, ......, fn−1}, and a corresponding question Q = {w0, w1, ......, wm−1} with associated SRL arguments S = {S0, S1, ......, SN−1}
where Si = {wi, wi+1, ......, wk}. All frames V = {f0, f1, ......, fn−1} are fed into an IMAGE ENCODER followed by temporal attention modeling to produce temporal-aware frame representations V
′ = {f ′0, f ′ 1, ......, f ′ n−1} ∈ Rn×d. Mean-
while, we use a TEXT ENCODER to obtain the representations of the question with its corresponding SRL arguments: Q
′ ∈ R1×d and S′ ∈ RN×d. We then perform multi-step reasoning in which we iteratively update the hidden state vector h with the visual information from frame representations based on the attention weights between them and the SRL arguments of the question. h is updated from the initial step h0 to the final step hT−1 where T is the total number of reasoning steps. Finally, we predict the most probable answer a based on hT−1.

heading: 2.1 Multi-step Reasoning
text: Before the first reasoning step, we initialize:
h0 = Attn(Q ′ , V ′ , V ′ ) (1)
j = argmax(AttnWeights(Q ′ , V ′ , V ′ )) (2)
where Attn serves as the q, k, v attention1 modeling (Vaswani et al., 2017) and j represents the
1In this work, we use a low temperature τ in the softmax to encourage the model to assign more attention weights to the most relevant frame.
index of the frame with the highest attention weight. In each specific reasoning step t, we firstly use ht−1 as the attention key to obtain the relevant SRL argument: S ′ t = Attn(ht−1, S ′ , S ′ ). Subsequently, we infer the next focused frame by:
V focus = Attn(rt, V ′ , V ′ ) (3)
where rt = g(ht−1, S ′ t). Finally, we update the hidden state vector ht−1 based on the currently focused frame (the frame with the largest attention weight):
ht = δ(ht−1, V focus) (4)

heading: 2.2 Retrospective-Prospective Reasoning
text: We propose a Retrospective-Prospective Reasoning mechanism for Eq.3 in order to explicitly decide whether the model should move to future frames (prospective reasoning) or move back to previous frames (retrospective reasoning). We obtain the retrospective frame V retro and prospective frame V prosp by:
V retro = ψ(g(ht−1, S ′ t), V ′ , RetroMask(j)) (5)
V prosp = ϕ(g(ht−1, S ′ t), V ′ , P rospMask(j)) (6)
where ψ and ϕ are MASKED ATTENTION that are used to obtain retrospective and prospective frames, g(ht−1, S ′ t) and V ′ serve as query and key, value respectively. RetroMask(j) means all frames after j (fi>j) will be masked whereas ProspMask(j) means that all frames before j (fi<j) will be masked. After obtaining V retro and V prosp we generate a probability:
p = σ(λ(V retro, V prosp)) (7)
If p is larger than a pre-defined threshold α, we update ht = δ(ht−1, V retro) ,otherwise we update ht = δ(ht−1, V prosp) as in Eq. 4. The index for the next-focused frame j is also updated accordingly. The reasoning process is shown in Algorithm 1.

heading: 2.3 Coverage Mechanism
text: We additionally propose to employ a coverage mechanism (Tu et al., 2016) to encourage the model to include as many SRL arguments as possible in the reasoning process. Specifically, we track the attention distribution Ct ∈ R1×N of ht−1 on all SRL arguments S
Ct = Ct−1 + AttnWeights([ht−1;Ct−1], S
′ , S ′ )
χ (8)
Algorithm 1: Multi-step dynamic retrospective-prospective reasoning with coverage mechanism V ′ = {f0, f1, ......, fn−1}: representations of video
frames Q ′ : question S ′ : SRL representations of Q T : reasoning steps χ : normalization factor α: threshold of the probability for using retrospective
frame h0 = Attn(Q ′ , V ′ , V ′ ) j = argmax(AttnWeights(Q ′ , V ′ , V ′ )) C0 = 0 for i in T do
S ′ i = Attn(hi−1, S ′ , S ′ , Ci−1) Ci = Ci−1 + AttnWeights(hi−1,S ′ ,S ′ ,Ci−1)
χ
V retro = ψ(g(ht−1, S ′ t), V ′ , RetroMask(j)) V prosp = ϕ(g(hi−1, S ′ i), V ′ , P rospMask(j)) p = σ(f(V retro, V prosp)) if p > α then
hi = δ(hi−1, V retro) j = argmax(ψ(g(ht−1, S ′ t), V ′ , RetroMask(j)))
else hi = δ(hi−1, V
prosp) j = argmax(ϕ(g(hi−1, S ′ i), V ′ , P rospMask(j)))
where χ represents the normalization factor.2 We obtain the weighted S ′ t by S ′ t = Attn([ht−1;Ct−1], S ′ , S ′ ) where we concatenate Ct−1 to ht−1 as an additional input to the Attn function for the purpose of informing the model to assign more attention weights to previously lessfocused SRL arguments, in order to improve the coverage for all SRL arguments.

heading: 2.4 Training Objective
text: For the answer prediction, we encode all answer options A = {a0, ......, aM−1} separately and then select the one with the highest similarity with hT−1. We optimize our model parameters θ using Cross Entropy loss:
J(θ) = − ∑
i
∑
k
log e(ak,hT−1)
∑M−1 j=0 e (aj ,hT−1) yi,k
(9) where  is the function measuring the similarity between answer candidate and hT−1, and yi,k represents the answer label for the i−th example - if the correct answer for the i−th example is the k−th answer then yi,k is 1 otherwise it is 0.
2In this work, we use the number of SRL arguments of the corresponding question as the normalization factor.

heading: 3 Experiments

heading: 3.1 Dataset
text: We employ a benchmark dataset for EVQA - TrafficQA (Xu et al., 2021) which contains 62,535 QA pairs and 10,080 videos. We follow the standard split of TrafficQA – 56,460 pairs for training and 6,075 pairs for evaluation. We further sample 5,000 examples from training data as the dev set.

heading: 3.2 Experimental Setup
text: We use CLIP ViT-B/16 (Radford et al., 2021) 3 to initialize our image encoder and text encoder. We evenly sample 10 frames from each video in the TrafficQA dataset. The SRL parser employed in the experiments is from AllenNLP (Gardner et al., 2018; Shi and Lin, 2019). We train our model over 10 epochs with a learning rate of 1 × 10−6 and a batch size of 8. The optimizer is AdamW (Loshchilov and Hutter, 2019). We set the maximum reasoning step T to 3 and we use a temperature τ of 0.2 in Attention modeling. The hyper-parameters are empirically selected based on the performance on dev set. There are two experimental settings for TrafficQA (Xu et al., 2021): 1) Setting-1/2, this task is to predict whether an answer is correct for a given question based on videos; 2) Setting-1/4: this task follows the standard setup of multiple-choice task in which the model is expected to predict the correct the answer from the four candidate options.

heading: 3.3 Results
text: The experimental results on the test set of TrafficQA are shown in Table 1, where we also in-
3https://openai.com/blog/clip/
clude the previous baseline models for EVQA.4 The results show that our proposed approach obtains accuracy of 43.19 under the multiple-choice setting, which surpasses previous state-of-the-art approaches including Eclipse (Xu et al., 2021), ERM (Zhang et al., 2022), TMBC (Luo et al., 2022) and CMCIR (Liu et al., 2022) by at least 4.5 points. Furthermore, our approach achieves an accuracy of 71.63 under Setting 1/2, outperforming previous strong baselines by at least 6 points. The results show the effectiveness of our proposed multi-step reasoning approach for event-level VideoQA.
Ablation Study We conduct experiments on the dev set of TrafficQA, investigating the contribution of both the retrospective-prospective reasoning and coverage mechanism on the performance of our proposed EVQA approach. The results are shown in Table 3, which reveals that multi-step reasoning is critical in terms of model performance while the coverage mechanism can provide additional, albeit less substantial, improvements.
Results by Question Type We take a closer look at model performance on different question types, e.g. reverse reasoning, counterfactual reasoning, etc. The results are shown in Table 2. They reveal that our proposed approach outperforms previous state-of-the-art models on all individual question types by a large margin with large improvements seen for introspection, reverse and counterfactual questions.
Effect of Reasoning Steps We study the effect of varying reasoning steps. The results are shown in Table 4. Increasing reasoning steps improves performance, especially from 1 step to 3 steps. Additionally, the performance (both Setting 1/4 and 1/2) is stable with reasoning steps exceeding three.

heading: 4 Conclusion and Future Work
text: In this paper, we propose a multi-step dynamic retrospective-prospective approach for EVQA. Our approach employs a multi-step reasoning model that explicitly learns reasoning based on the semantic connection of the SRL structure of a question and corresponding video frames. We additionally proposed a coverage mechanism to improve the coverage of SRL arguments in the reasoning process. Experimental results show that the proposed
4Some of the baseline results are taken from Xu et al. (2021).
approach obtains superior performance compared to that of state-of-the-art EVQA models.

heading: Acknowledgements
text: This work was funded by Science Foundation Ireland through the SFI Centre for Research Training in Machine Learning (18/CRT/6183). We thank the reviewers for helpful feedback.
Limitations
This papers focuses on a variety of VideoQA - event-level VideoQA, we only incorporate event information from the question (textual) side as we think that parsing video frames is inaccurate and could introduce unexpected errors, we should also explore how to inject event-level information from visual side in the future with more competitive visual parsing models. Our experiments are only conducted on one dataset due to resource constraint, we should also conduct experiments on
more datasets to verify the effectiveness of our approach.

heading: Future Work
text: Limitations
This papers focuses on a variety of VideoQA - event-level VideoQA, we only incorporate event information from the question (textual) side as we think that parsing video frames is inaccurate and could introduce unexpected errors, we should also explore how to inject event-level information from visual side in the future with more competitive visual parsing models. Our experiments are only conducted on one dataset due to resource constraint, we should also conduct experiments on
more datasets to verify the effectiveness of our approach