Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Authors: Weijiang Yu
Haoteng Zheng
Mengfei Li
Lei Ji
Lijun Wu
Nong Xiao
Nan Duan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video Question Answering
Multi-view Learning
Data-effeciency Reasoning

Abstract: Recent advances in the video question answering (i.e., VideoQA) task have achieved strong success by following the paradigm of fine-tuning each clip-text pair independently on the pretrained transformer-based model via supervised learning. Intuitively, multiple samples (i.e., clips) should be interdependent to capture similar visual and key semantic information in the same video. To consider the interdependent knowledge between contextual clips into the network inference, we propose a Siamese Sampling and Reasoning (SiaSamRea) approach, which consists of a siamese sampling mechanism to generate sparse and similar clips (i.e., siamese clips) from the same video, and a novel reasoning strategy for integrating the interdependent knowledge between contextual clips into the network. The reasoning strategy contains two modules: (1) siamese knowledge generation to learn the inter-relationship among clips; (2) siamese knowledge reasoning to produce the refined soft label by propagating the weights of inter-relationship to the predicted candidates of all clips. Finally, our SiaSamRea can endow the current multimodal reasoning paradigm with the ability of learning from inside via the guidance of soft labels. Extensive experiments demonstrate our SiaSamRea achieves state-of-the-art performance on five VideoQA benchmarks, e.g., a significant +2.1% gain on MSRVTT-QA, +2.9% on MSVD-QA, +1.0% on ActivityNet-QA, +1.8% on How2QA and +4.3% (action) on TGIF-QA.

Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Weijiang Yu', Haoteng Zheng', Mengfei Li', Lei Ji2, Lijun Wu?, Nong Xiao Nan Duan? School of Computer Science and Engineering, Sun Yat-sen University
Microsoff Reasearch Asia
wei aiigiangyu&@gmal com, xiaon momom60mail.. ees.mmmlll...eeee...e edu.
(zhenght5, 1imf33/ @mmai12 sysu edu.
(leiji, Lijun.wu. wu, nanduan @microsoft. com
Abstract
Recent advances in the video question answering (i.e., VideoQA) task have achieved strong success by following the paradigm of fine-tuning each clip-text pair independently on the pretrained transformer-bes based model via supervised learning. Intuitively, multiple samples (i.c., clips) should be interdependent to capture similar visual and key semantic information in the same video. To consider the interdepen- dent knowledge between contextual clips into the network inference, we propose Siamese Sampling and Reasoning (SiaSamRea approach, which consists of siamese sampling mechanism generate sparse and similar clips (i.e., siamese clips) from the same video, and novel reasoning strategy for integrating the inter- dependent knowledge between contextual clips into the network. The reasoning strategy contains two modules: (1) siamese knowledge generation to learn the inter-relationship among clips; (2) siamese knowledge reasoning to produce the refined soft label by propagating the weights of nnter-relationship to the predicted candidates of all clips. Finally, SiaSamRea can endow the current multimodal reasoning paradigm with the ability of learning from inside via the guidance soft labels. Extensive experiments demonstrate our SiaSamRea achieves state-of- the-art performance on five VideoQA benchmarks, e.g., significant +2.1 % gain MSRVTT-QA. +2.9% MSVD-QA, +1.0% ActivityNet-QA, +1.8 % on How2QA and +4.3 % (action) on TGIF-QA.
Introduction
By inferring the correct answers for video-based questions, video question answering (VideoQA) has attracted increasing research attention due its huge application potential, as fundamen- reasoning.
Dense Clip
Sparse Clip
Siamese Clips
tal technique for vision-to-langen The task involves acquisition and manipulation of spatio-temporal visual representations guided by the compositional semanties of the linguistic clues [32 [15
[a] Dense Sampling (b) Sparse Sampling
(c) Siamese Sampling
Figure 1: Different sampling mechanisms for video frames. (a) Traditional methods use dense clip features from full-length videos. (b) A re- cent approach [22 suggests sparsely sampled clips for end-to-end learning. (c) Our siamese sampling to generate similar semantic clips.
21 34]. Existing works can roughly be divided into two aspects. One aspect to ex- plore powerful multimodal transformer-basee net- work [22 2 34 45 trained on large-scale datasets
(e.g., COCO Captions
3 and HowTo100M, (29]).
*Corresponding Author: xiaon6 @ www.aaai......u..mmmmm
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

knowledge
Guideraa
Beasnning
Panchor
[p,, Pa Pal
Prediction
Prediction
Clessifler
Classifier
Stamesse
Know wage
Classifier
Classifier
f anchor
f
f
Cross-modal
Cross-modal Model
Cross-modal
Cross-modal Model
Model
Stop Gradient
t
V
V anchar
Encoder
Video
Text Encoder
Video Encoder
Text Encoder
Video Encoder
Encoder
Video
Text Encoder
Sampling
Frame
Word Embedding
Sparse Frame Sempling
Embadding
Sampling
Sampling
Stameee
Embedding
Word
Video
Text
Video
Video
Text
(x) Traditional Multimodal Learning
(b/ Sparse based Multimadal Learnnng
(c) Self-drivo driven based Multimodal Learning
Figure 2: (a) Traditional multimodal learning uses dense sampling for video frames and extracts individual clip-text features via offiin encoders. (b) Sparse based multimodal learning benefits from modeling. (c)
sparsely sampled clips (independent) and raw text tokens for based multimodal learning utilizes cross-relationship between anchor clip (Vanchor) and siamese clips (V1, V2, Va) as knowledge lead the model to create soft labels for self-supervised learning. Note that the modules in (c) using same color mean the weights are shared.
The other aspect aims at exploring the structure reasoning for semantic alignment between vision and language (c.g.. Hicrarchical Reasoning [21]. Heterogeneous Graph Aligmment [155 46 and Object Relation Reasoning [6].. Both of them solely consider each clip-text pair separately and ignore the correlation between contextual clips in the same video.
By further analyzing the existing multimodal learning paradigm and showing their difference in Figure [ we observe that current methods suffer from key drawback: every clip-text pair is regarded individual and independent during the training. Such drawback overlooks the rich interaction of contextual clips from the same video? We believe that the internal interaction information from same videos can be helpful for further enhancing the network learning. Hence, provide remedy to this dilemma, we present self-driven Siamese Sampling and Reasoning (SiaSamRea) framework learning from inside, by using the internal contextual semantics of interdependent video-aware data (e.g., clips) from the same video in the training process.
First of all, SiaSamRea (in Figure consists of two key parts: (1) siamese sampling shown in Figure (c) to extract multiple similar clips from the same video, which is motivated by ClipBert [22]; (2) reasoning strategy named self- based multimodal learning as shown in Figure 2 (c). The strategy contains two modules: (i) a siamese knowledge generation module to calculate the correlationn matrix between the anchor clip-text pair (e.g., fanchor) and siamese clip-text pairs (e.g., fū, f2, fa); (61) a siamese knowledge reasoning module to produce soft labels for self-supervised refinement during the training. Finally, the labels are applied as auxiliary training supervision to enhance the network.
Compared with previous sampling mechanism, our siamese sampling as shown in Figure D (c) not only is sparse but also is able generate multiple similar clips for constructing their internal relationships. Specifically, our siamese sampling captures clips at different start frames with same interval time in the same video, which constrain the global semantic of each clip to be similar (i.e., each clip can represent the consistent video content from global perspective).
Furthermore, to fully utilize the siamese clips, we explore a new reasoning strategy namely self-driven based multimodal learning as shown Figure (e). There are three steps in the strategy for using the interaction of internal clips from the same vidco into the training. At the 1s step, the anchor clip and siamese clips are obtained by using sparse sampling and siamese sampling, respectively. Then the two types of clips individually cooperated with the text are fed into the model to extract clip-text features, including anchor clip-text feature (e.g., Vanchor ) and siamese clip-text features (e.g., V1, V2, V3). At the 2nd step, the internal contextual interaction between the anchor clip-text feature and siamese
2 We call the contextual clips from same video as internal clips containing anchor and siamese clips The interal contextual interaction is regarded as siamese knowledge in our paper.

---

clip-text features is calculated via our siamese knowledge generation module. At the 3rd step, siamese knowledge reasoning module is proposed use the siamese knowledge applying to several predicted candidates (e.g., Panchor; PI, Pt P2, Pa) for adaptively reasoning out the refined soft label. Here, the siamese knowledge is applied for adaptively reasoning. Because is hard to distinguish which candidate is critical to the accuracy from several predictions during the training. Finally, we use the refined soft label to further distill our model for high-quality representation generation.
Our contributions are three-fold: (i) We propose novel end-to-end framework named SiaSamRea for learning from inside on VideoQA task, by using siamese sampling and reasoning to integrate the interdependent semantics of clips from the same video into the training process. (ti) A novel reasoning strategy is carefully designed for building the soft guidance from the interdependent knowledge between internal clips, which consists of a siamese knowledge generation module and siamese knowledge reasoning module. (iii) Experiments on five commonly-used VideoQA benchmarks show the superior ability of our SiaSamRea and demonstrate the effectiveness of our proposed components. Not that our method only teaches the network with interdependent knowledge during the training. which does not bring any extra burden (e.g., computation, memory and parameters) in the inference.
2 Related Work
Visual Content Modeling on VideoQA. Video Question Answering is a task aiming to answer the given question concerning video content. Some current works extract generic visual appearance and motion features to represent video contents and design different attention mechanisms to integrate these features, like question-routed attention 38 14
and co-attention p These methods mainly focus on the holistic understanding of video contents, which easily neglect the meaningful details of local clips. Some research pays attention to structure reasoning for semantic alignment [6]] [I]
[7]. Jiang et al. [15) proposed to build the correlation between the inter- and aitra-modalities for cross-modal learning via a heterogeneous graph alignment framework. Huang et al. [ presented novel location-awe aware graph convolution network to mine the structural representation between the location and relation among visual objects. Le et al. [2T] demonstrated the effectiveness of hierarchical video features for VideoQA by developing a general- purpose neural reasoning unit. These works aim to handle fine-grained visual entity (e.g., object) and multimodal hierarchy but ignore the correlation between contextual clips in the same video.
dhetrained-innttuning Paradigm on VideoQA. Recently, the paradigm of pretrained fine-tuning has made significant progress in many multimodal tasks, such as text-video retrieval 33 visual question answering M. image captioning
44 video captioning 39] and video question answer-
ing [23 Thanks to the strong success of transformer-
[37 language pretraining 6 27
35| has shown promising improvement on the video- langu-ang tasks, especially on VideoQA task. For example, Zhang et al. (50) proposed to enhance the visual representation based on the OSCAR (26 by in- corporating more visual data during the training. Lei et al. [22 solved the offline encoder in the netrained-finetuninn pipeline by adopting a sparse training and dense inference. To promote the development of large-scale multimodal learning, Seo et al. (34] proposed a new dataset and task for future utterance prediction based on the video-text inputs. The multimodal model trained on their new dataset also performed well on the VideoQA task. It can further prove the transfer ability of the pretrained- pretrained-finetuinng paradigm. All approaches as mentioned-above on VideoQA are benefited from the powerful pretrained backbone with large-scale clip-text pairs, while each clip-text pair is independently encoded into the network rather than mining their contextual knowledge in the same video. In this paper, we argue that the rich contextual information among clips should be valuable and can well enhance the network. Hence, we propose a siamese sampling and reasoning method for learning from internal clips in the same video as well as bringing additional burden in the inference, which consists of a siamese sampling mechanism and a reasoning strategy.
and image-text pretraining [36 5 B1) the video-text pretraining [24 29 51
3 Methodology
Our Siamese Sampling and Reasoning (SiaSamRea) method shown in Figure The SiaSamRea can be roughly divided into three parts based on the learning pipeline: (1) Clip Sampling; (2) Feature Extraction: (3) Reasoning Strategy. The clip sampling aims to get the anchor clip and siamese clips by sparse sampling and siamese sampling, respectively. The feature extraction is composed of a video
3

---

Video Frames
Anchor Clip
Video
V
Multimodal
FC
Classifier
Transformer
Guidance
Fircher
Punchr
Share Weights
Share Weights
Text
Stamese
Siamese
Encoder
Knowledge
Knowledge
Generation
Reasoning
Slamese Clips
Textual Inputs
fufufs.
PuPpPPs
Video
N,N,
Multimodal Transformer
Encoder
FC
FC
Classifier
Siamese Sampling
Reasoning Strategy
Figure 3: Overview of our Siamese Sampling and Reasoning (SiaSamRea) applied to VideoQA. Given the dense video frames, the anchor clip and siamese clips first to be extracted by sparse sampling and siamese sampling. Then a video/text encoder and a multimodal transformer are utilized to generate multimodal features (fanchor: f1, f2, fz). Next, siamese knowledge generation module is proposed to produce contextual relationship between anchor clip and siamese clips from the same video. Finally, the soft label used for distilling the whole network, can be obtained via siamese knowledge reasoning module by inferring the siamese knowledge on several candidates (Panchor P1, P2, pa). Note that we only need the network fed with anchor clip for inference.
encoder, a text encoder and a multimodal transformer, which encodes the multiple clip-text pairs as semantic representations. The goal of the reasoning strategy is generate refined labels for distilling the model learning, which can be achieved by using the siamese knowledge reasoning module to propagate the siamese knowledge (i.e., interdependent relationship among pairs) from the siamese knowledge generation module into the predictions of multiple clip-text pairs. Our contribution mainly single framework, the siamese sampling mechanism
focuses on the design of reasoning strategy and innovative modules (e.g., siamese knowledge generation and siamese knowledge reasoning), which are unveiled and discussed in details in the following sub-sections.
3.1 Preliminary
To simplify our method, we unify some symbolic notations in this section. Given the dense video frames F, we seek to construct the anchor clip Canchor and siamese clip set Csiamese = [c N-1 where the N means all clips number. Each clip is uniformly sampled frames and obtains B feature maps by using the video encoder to encode the clip. The encoded features of anchor clip and the siamese clip set are denoted as Vanchor and yssiames = [ N-1
correspondingly. Taking the encoded text representation from the text encoder concatenated with the clip representation from the video encoder as input, multimodal transformer is applied to produce the clip-text feature. Each anchor elip-text feature is indicated fanchor and the associated siamese clip-text feature set indicates Eviamese
Ji=1
{ Similarly, their predictions are defined as Panchor and Psiamese WP respectively. The is a vector with K dimension according to the classification number.
3.2 Siamese Sampling
In this section, we introduce our siamese sampling shown in Figure (i), which is inspired by the spare sampling in ClipBert (22). First, we get an anchor clip Canchor by sparsely and uniformly sampling B frames from dense video frames F. The length of typically much larger than B. We randomly decide the starting sampling index of the anchor clip. Then our siamese sampling captures siamese clips different starting indexes but nearby the index the anchor clip in the same video. We use the same interval frame for all clips. There are some constraints for selecting the starting sampling index of the anchor clip. should be the first third of the video to guarantee equilibrium of sampling. The "nearby" means the index of the Siamese clip should be next to the index of the anchor clip. The pace of sampling is consecutive before or after the anchor index. After obtain some siamese clips, which have similar global vidco semantics
siamese sampling,

---

including the anchor clip. Then we feed these clips into the video encoder sequentially as shown in Figure to get the visual feature. The video encoder and text encoder are similar with the encoders in the ClipBert [22].
3.3 Reasoning Strategy
that directly utilized the clip-text features from multimodal transformer to predict the final results, we propose a novel reasoning strategy named self-driven multimodal learning by exploring how integrate knowledge between different clips (i.e., siamese clips) in the same video with single network, which consists of a siamese knowledge generation module and siamese knowledge reasoning module.
Different from previous methods 442 22
3.3.1 Siamese Knowledge Generation
Intuitively, clips with high visual similarities expected to have more consistent predictions on their predicted class probabilities, regardless their ground-truth labels. In our solution, similar clips' knowledge is systematically aggregated and integrated provide better soft labels.
We propose to propagate and integrate knowledge among clip-text pairs on-the-fly in terms of their multimodal feature similarities. Given set of N clip-text pairs from the same video and feature extractor F under training, we first estimate the samples' pairwise similarities by the dot product of their encoded representations with the current network. Such similarities can be stored in an interdependent matrix A E IR N as
Ai.j) (FF()) (FFE)))))))(((((())
where j the indices of samples in video and denotes the 82-norm function. N is the number of samples. To avoid reasoning in the self-loop reinforcement, we discard the diagonal entries from A by A A (1 I). where I is an identity matrix and @ denotes the Hadamard product. Subsequently, we normalize each row of the interdependent matrix so that Σ Aii,j = 1 for
all i, while keeping the diagonal all zeros, i.e., A i) O. The normalization can be formulated as softmax function over each row of the matrix A
(x,(A(i,j) A(i Ejtt exp A (i.j))
Vi NJ.
where the jii indicates the normalized interdependent knowledge between the i-th sample and j-th sample in the same video. The whole interdependent knowledge of all sampling clips denoted as À is also called as siamese knowledge in this paper. Due to the knowledge mainly calculated by our siamese samples.
3.3.2 Siamese Knowledge Reasoning
We denote the predicted probabilities of samples within a video as P = .p......pN] E RNK, which satisfy ΣK ((i, k) 1, Vi. For the i-th sample in the video, there are K-class candidates. We would like to adaptively propagate and merge the other samples' predictions to create a better soft label for it based on the inter-sample affinities, which can be formulated as
p. A( - (())P
(3)
where P. is the propagated probability vector for the i-th sample and can serve as the refined soft labels. Intuitively, if the i-th sample and the j-th sample are similar with a high interdependent value (i..), the prediction P, would have a larger weight to be propagated to p;- Motivated by the graph convolutional network (18), module adaptively propagate the predictions between all the samples (i.e., clip set) in a video in parallel, which can be formulated as
P WAP,
(4)
where the W E RIXN is learnable matrix and P E R is our soft label. To avoid propagating and integrating noisy and unexpected predictions too much, we produce the soft learning targets Q as
"o simplify the expression, the "clip-text pair" denotes "sample" in Sec. 3.3
5

---

weighted sum of the initial probability matrix P and the propagated one AP,
WW (1-
(5) k=1 Q(k) - = 1, R The W1 and W2 are
where E [0. 1] is the weighting factor and Σ Σx trainable weights. With the above formulations, the knowledge between samples within the same video can be propagated to each other and integrated for one iteration.
3.4 Optimization
Open-ended VideoQA Setting. Open-ended question setting is to choose one correct answer from a predefined answer set Ω which can be seen as a multi-label classification task and trained with the cross-entropy loss function. We feed the visual representation and question representation q into our framework denoted as dg to compute prediction probabilities:
p (v q). R!
Multiple-choice VideoQA Setting, Multiple-choice question setting is to choose one correct answer from M candidates. In this case, first formulate the answer representation of the m-th candidate as am- Then, we feed the visual representation, question representation and answer representation into our model ouput the m-th answer score, which formulated as
8m (v q am), 1 S S M,
where the score of the correct candidate is the positive score s", and the rest scores are negative scores denoted as (sf, ..., 2)). During training, we utilize the summed pairwise hinge loss
1= maz(0, 1 (s" s))) between the positive score and each negative score to train our model.
M-1
terms for our final objective function L. One term denoted as Sisemese comes from the loss between our soft label and the prediction result Panchor from the anchor clip-text pair. We utilize the cross-entropy loss function to compute the loss Lisamesee- The other term is to optimize the training process between predictions and the ground-truth, which is commonly in the previous works. Note that the last term may be different depended the task setting, we uniformly formulate the loss function associated with the ground-truth as Cgz. Finally, our model can be finally optimized by
Objective Function. There
-= L= + Leghs
(8)
where a is hyper-paramest term to adjust the balance of the two losses.
4
Experiment
In this section, we first introduce the data that we use in Sec. 4.1 and details of our experiments in Sec. 4.2 Then we validate the effectiveness of our proposed components in Sec. 4.3 which is
followed by the comparison with other methods in Sec. [4.4, Finally, in Sec. 4.5) some visualization
results are shown to qualitatively analyze the benefits of our siamese clips.
4.1 Evaluation and Datasets
We evaluate our proposed VideoQA method and compare it with other state-of-the-a methods on five VideoQA datasets, which are widely used and accepted for academic video question answering, including the open-ended setting and multiple-choice setting. We follow the previous evaluation protocols for open-ended settings [21 (48) and utilize a fixed vocabulary training answers. For the multiple-choice evaluation setting, we follow the previous methods [21] to use the same dataset split setting. Excepting for the repetition count task in the TGIF-QA [3] where the Mean Squre Error (MSE) is excavated, we use top-1 accuracy to be the evaluation metric for all experiments.
We carry out our SiaSamRea on three open-ended VideoQA datasets, including MSRVTT-QA (8), MSVD-QA [8 and ActivityNet-ON QA [48]. The MSRVTT-QA has 10K videos and 244K question- answer pairs. The size of the predefined answer set 1000. Compared with the MSRVTT-QA. the MSVD-QA is smaller, which has 2K videos with nearly 51K question-answer pairs. The MSRVTT- QA and MSVD-QA contain five question types like what, who, how, when and where. ActivityNet- QA has 5.8K videos and 58K questions, which is repurposed from ActivityNet Captions [1]]

---

Furthermore, we validate our method on a multiple-choice VideoQA datasets, such as How2QA [24]. The How2QA has 9K videos tailed with 44K question numbers. where each question associated with one correct and three incorrect answers. Besides, we train and test our model on the TGIF-QA [3] which is a large-scale VideoQA dataset containing 72K animated GIFs and 165K question-answer pairs. There are four tasks defined on this dataset: (x) Repetition Count is an open-ended QA task to
count the number of the repetitions of an action; (ti) Repeating Action is similar to the repetition
count task but following the multiple-choice setting: (iii) State Transition is multiple choice task for identifying the transition between two states; (iv) Frame QA is open-ended to find the sufficient frame to answer the questions.
4.2 Implementation Details
We implement our proposed SiaSamRea via PyTorch [0), a well-known open- open-source deep learning framework. For being fairly comparable with other methods, we conduct the following training and testing settings. Unless otherwise noted, settings are the same for all experiments.
Training. We use the pretrained model trained on the COCO Captions [3] and Visual Genome Captions [20] from ClipBert [22] as our initial weights of our SiaSamRea. To obtain more Vi related pretrained weights, we present to train task-related data during the pretraining. Based on the initial weights from the ClipBert, we perform the clip-text pretraining on TGIF-QA [13] optimized by adaptively using aforementioned objection function (i.e., Eq. 8) according to different task settings (i.e.. open-ended and multiple-choiee tasks). There a total of 71739 videos in the TGIF-QA dataset. Because our proposed method aims to model the long-range contextualized knowledge via Siamese samples. The videos with less than 40 frames do not meet our requirements in the training. We filter out less than 40 frames of video during the pre-training to remain long-range temporal videos. In the TGIF-QA QA dataset, the percentage of videos with less than 40 frames 56% (40333/71739). Hence, we use the remaining 3 31006 TGIF videos for our pre-training, which is less than the ClipBERT. The ClipBERT uses a total of 71739 TGIF-QA videos to finetune the four tasks, which different from our pre-training on the TGIF-QA dataset. Because the video length is various, we set the number of frames in each clip from to 16 according to the video length. We set the batch size to per GPU and use AdamW (28) optimizer with initial learning rate 0.0005. The learning rate warm-up strategy is adopted over the first 10% training steps followed by linear decay to zero. We terminate the pretraining when it reaches minimal learning rate (i.c., 0.00001) and its validation performance stabilizes (i.e., reaches convergence). The balance term is o=1.0, the integrating weight is w=0.5. We conduct all experiments using 16 NVIDIA V100 GPU cards.
For the downstream fine-tuning, we apply the same training and optimizer configurations as mentioned-above. Since the downstream datasets are various and in multiple domains, we use dataset-specific learning rates, training epochs and clip numbers based on validation performance. Note that we do not need to excavate additional fine-tuning on the TGIF-QA dataset.
Testing, We strictly follow the test split setup in each dataset and validate the learning outcomes after each learning epoch. Then we report the best metrics (top-1 accuracy) as our results.
4.3 Ablation Studies
We conduct comprehensive ablation studies con- cerning various aspects of our SiaSamRea's 's de- sign in this section. In Table [ we set the network training without using the Lviamese as baseline. Compared with the baseline, our method only using the siamese knowledge reasoning module (w/ SKR) by replacing the siamese knowledge generation with average op- eration, can apparently boost the accuracy by more than 4% from baseline on How2QA and MSVD-QA. can demonstrate the effectiveness of our siamese knowledge reasoning. It also validates our assumption that multiple clips from same videos can really bring some strong knowledge to en- hance the network ability. When we add the siamese knowledge generation module (w/ SKG+SKR), our method can reach a higher performance to 84. 1 1% and 45.5%, which can demonstrate the ef- fectiveness of building the interdependent knowledge (i.e., siamese knowledge) for integrating the samples. It can also prove the adaptively reasoning by our siamese knowledge is better than the purely
Table 1: Ablation studies on How2QA and MSVD- QA datasets. The SKG and SKR separately indi- cate siamese knowledge generation and siamese knowledge reasoning.
Methods baseline
How2QA
MSVD-QA
79.1
39.4
w/ SKR
83.0
44.7
w/ SKG SKR
84.1
45.5
7

---

average operation. We think that the siamese knowledge not only serves as the knowledge-rouse routed representation, but also implicitly constrains the semantic consistency of clips in the space of clip-text features.
To explore the effect of the number of siamese clips, we compare our method with various number of siamese clips, which results can be shown in Table 2 Note that our siamese sam- ples are serving for the interdependent knowl- edge extraction. Intuitively, more samples can aggregate richer knowledge. When adding the number of siamese samples from 1 to 12, our method promotes the accuracy from 79.6% to 84.4% on How2QA and 39.9 59.9%-445.7 on MSVD-QA. Such improvement can demonstrate our intuition that more siamese samples can bring richer knowledge, which makes our network benefited from it. Although the accuracy increasing with the number of siamese clips, we observe that the improve- ment from the number 8 to 12 is slight. We think the reason is the saturation of knowledge, i.e., the model has enough knowledge to learn the task on the two datasets. Hence, it is almost meaningless to purely increase the siamese samples. To balance the training time and accuracy, we select the 8 siamese samples in our final version.
Table 2:
The effect of the number of sampling siamese clips on How2QA and MSVD-QA.
Methods How2QA MSVD-QA
2
8
12
79.6
80.5
81.9
84.1
84.4
39.9
41.3
42.7
45.5
45.7
The effect for WI and W2 in Eq. 3 We have conducted experiments on the MSVD-QA dataset to analyze the effect of these learnable weights (WI and W2). Our SiaSamRea without WI performs 44.9% on MSVD-QA, which heavily drops 0.6% performance compared with our whole SiaSamRea (45.5%). One possible reason that the W1 can make the propagated prediction matrix learnable and adaptively propagate the predictions between all the samples. When performing our model without W2, it achieves 45.2 5..2%. The performance without W2 is slightly decreasing ( 0.3%). When comes to method without using WI and W2, it gets 44.5%. As we can see, the WI and W2 both can bright improvement. Hence, it can further demonstrate the importance and necessity for the generation of expected soft labels by correctly merging several predicted candidates.
Table 3: Comparison with state of the art on MSRVTT-QA and MSVD-QA (top-1 accuracy).
The effect for multiple copies of anchor clip. We replace our Siamese clips with multiple copies of anchor clips to analyze the effect of SKG. Since we used 8 siamese clips in our paper, we replaced the siamese clips with 8 copies of anchor clips during the training. We execute this ablation study on the MSVD-QA dataset. This ablation study achieves 41.3%. It promotes the baseline (39.4%) with 1.9% performance. It can validate the power of the ensemble way. But only using the anchor clips for ensemble can't bring visually contextual- ized knowledge. We skillfully exploit the visually contextualized knowledge via the siamese clips, which obtains 45.5% accuracy. Hence, the im- provement of our method mainly comes from the SKR and SKG by using Siamese clips rather than the simple ensemble.
Methods
MSRVTT-QA MSVD-QA
E-SA [38]
29.3
27.6
ST-TP 13
30.9
31.3
AMU 38
32.5
32.0
Co-mem 9
32.0
31.7
HME 7
33.0
33.7
LAGCN
34.3
HGA [15
35.5
34.7
QueST 14
34.6
36.1
MINOR [16
35.4
35.0
TSN [40]
35,4
36.7
HCRN 21
35.6
36.1
Clip-BERT 22
37,4
SSML
35.1
35.1
CoMVT
39.5
42.6
SiaSamRea (Ours)
41.6
45.5
4.4
Comparisons with State-of-the-Arts
Comparisons on MSRYTT-QA and MSVD- QA. Table reports the comparison with previous methods on MSRVTT-QA and MSVD-QA datasets. Our SiaSamRea achieves the best performance on both datasets. Specifically, compared with the methods using large-scale pretrained data (i.c., Clip-BERT, SSML and CoMVT), our method consistently and considerably outperforms them by 2.1% and 2.9% top-1 accuracy on MSRVTT-QA and MSVD-QA when compared with CoMVT, which was previously the best performer. When comes to other methods (e.g., HCRN, TSN etc.) without using additional pretrained data, our model can significantly boost the accuracy by around 6% accuracy on MSRVTT-QA and a large gain of 9.4% on MSVD-QA compared with HCRN. These promising results can validate the effectiveness of using pretrained data, It also supports the feasibility of distilling the interdependence of multiple clips with our reasoning strategy on the open-ended VidcoQA task.
8

---

our SiaSamRea consistently out- performs state-of-the-arr models Comparisons on ActivityNNet-QA and How2QA. We now move to the evaluation on ActivityNet- QA and How2QA, which results are reported in Table 4 In particular, our method improves over the recent CoMVT approach that has been pretrained on HowTo100M [29] dataset. It can effectively demonstrate the advantages of apply- ing related-data (c.g.. TGIF-QA) on the pretrain- ing, which can perform well to transfer to related tasks. The ActivityNet-QA is so challenging due to the complex events and backgrounds in the videos. Our SiaSamRea still achieves a new state- of-the-art performance (39.8%) on ActivityNet- QA by learning from internal clips. These strong results show the importance and powerful ability of our proposed learning strategy. In other words, SeaSamRea can provide another way handle the complex situation by mining the intrinsic information like internal relationship to parse the various temporal scenes from multiple samples. On the multiple-choice task, the ability of our method gets great scores (84.1%) 1%) on How2QA dataset as validating the availability of transferring our method to the other video question answering settings.
Table 4: Comparison with state of the art on ActivityNer-QA and the public val set of How2QA (top-1 accuracy).
Methods
ActivityNet-QA How2QA
E-SA 48
31.8
MAR- VQA
34.6
HERO 24
74.1
CoMVT 33
38.8
82.3
SiaSamRea (Ours)
39.8
84.1
Table 5: Comparison with the state-of-the-ar methods on TGIF-
Comparisons TGIF-QA. To further demonstrate the advantage of SiaSamRea to infer more vari- ous and difficult scenarios, we es- timate it the TGIF-QA dataset. The experimental results are sum- marized in Table 5 It is clear that
QA dataset. For count, the lower the better.
Methods
Action
Trans.
Frame
Count
VIS+LSTM (agg)
32
46.8
56.9
34,6
5.09
VIS+LSTM (avg)
32
48.8
34.8
35.0
4.80
VQA-MCB (agg)
8
58.9
24.3
25.7
5.17
VQA-MCB (avg)
29.1
33.0
15.5
5.54
CT-SAN 47
56.1
64.0
39.6
5.13
ST-TP 13
62.9
69.4
49.5
4.32
on four reasoning tasks. The four tasks are requiring the strong tem- poral reasoning especially for the action and transition task. Hence,
GR-ATT 38
68.8
73.9
53.0
4.32
Co-mem
68.2
74.3
51.5
4.10
PSAC 255
70.4
76.9
55.7
4.27
STA I0
72.3
79.0
56.6
4.25
they really require the model to consider the contextual depen- dence of clips. Particularly, our method increases overall accuracy on the publicly test set by 4.3% 75.4%~779.79 compared with the best result on repeating ac- tion task, 3.9% (81.4%++85335)
MiNOR 16
72.7
80.9
57.1
4.17
HME 7
73.9
77.8
53.8
4.02
HCRN 21
75.0
81.4
55.9
3.82
ClipBERT
75.1
80.8
56.3
4.07
HGA [1
75.4
81.0
55.1
4.09
SiaSamRea (Ours)
79.7
85.3
60.2
3.61
FrameQA task and decreases error by 0.21 (3.82+3.61) on repetition count task. Thanks to the siamese sampling and reasoning, our method effectively capture the consistent video content from a global perspective. Because SiaSam- Rea can distinguish the global semantic of a video by multiple views (i.c., siamese clips). which can bring abundant information from inside that may be absent in other clips to help the network learning the contextual relationship.
state transition task, 3.1% (57.1%-660.25%
To fairly compare the effect of ClipBERT with proposed method, we conduct extra experiments to finetune the CRT on the filtered data and test it on the four tasks. We mark the ClipBERT finetuned our filtered data as ClipBERT* The ClipBERT* gets 75.1 % on Action, 80.8% on Trans., 56.3% on Frame and 4.07 on Count. As we can see, the ClipBERT* can achieve comparable results compared with other SOTA methods. It can prove the effectiveness of the pre- treitriined weights from COCO Captions and Visual Genome Captions. However, our proposed method still outperforms the ClipBERT* on four tasks based on the same filtered data (79.7% on Action, 85.3% Trans., 60.2% on Frame and 3.61 on Count). It can further demonstrate the superior ability of our idea by using the Siamese clips during the training. Our proposed method achieves better performance with less data. also validates the huge potential of inside data once there a suitable way to explore them.

---

Q: What perched on a branch of tree bit a tail of another animal? A: Bird
Anchor
Clip
Siamese Clips
Figure 4: The examples to show the benefits from the siamese clips. It is hard to discriminate the visual content "bird" from ambiguous parts (red circle) in anchor clip. It much easier distinguish "bird" by the assistance of siamese clips that contains more complete visual content (green circle).
4.5 Visualization
In Figure 4 we show training example to qualitatively analyze what kind of knowledge does the siamese clips bring to the anchor clip. is unavoidable to lose some visual contents when sparsely sampling from the video. For example, the visual content "bird" in anchor clip is almost absent. However, we can obtain the absent content from the siamemse clips due different sampling indices and duration. Hence, our siamese samples can help the model to correctly and stably train on the sparse samples as well as avoiding confusing predictions due to the incomplete visual information.
5 Conclusion and Boarder Impact
In this paper, we propose endow the current multimodal reasoning paradigm with the ability of learning from inside on the VideoQA task via Siamese Sampling and Reasoning (SiaSamRea), which contains two key parts: (1) siamese sampling to produce some sparse clips with similar semantics in the same video; (2) reasoning strategy to distill the interdependent knowledge between clips into the network. The reasoning strategy is composed of modules: (i) siamese knowledge generation to implicitly aggregate the inter-relationship of clips from the same video; (ii) siamese knowledge reasoning to infer soft label by using the predicted candidates of all clips and their inter-relationship Our proposed SiaSamRea finally can be jointly evolved by the soft label guidance and ground truth, which is evaluated on five VideoQA datasets demonstrating state-of -fe----ttthhtte art performance.
This work analyzes an interesting problem of how to learn from inside on the video question answering. Different from current methods mainly look for insights from outside data and powerful pretrained models. Our work proposes a totally different insight called learning from inside. In this paper, we observe the potential power of clips in the same video. Intuitively, the multiple clips should contain rich interdependent knowledge which is ignored by current ad advanced methods. To the best of our knowledge, we are the first one present learning from inside by using the knowledge between internal clips to assist in evolving the network training. Moreover, our method only teaches the network with interdependent knowledge during the training, which does not bring any extra burden (e.g., computation, memory and parameters) in the inference. Hence, it is light way to easily equip our method with other suitable networks on different multimodal tasks. We hope that our work will increase interest in the exciting field of learning from visually contextualized knowledge.
6 Acknowledgments
We thank all reviewers for providing the thoughtful and constructive suggestions. This work was supported in part by National Natural Science Foundation of China under Grant No.U1811461, in part by Natural Science Foundation of Guangdong Province, China under Grant No. 82030312002. and in part by the Major Program of Guangdong Basic and Applied Research No.2019B030302000
10

---

References
[1] A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Parikh, and D. Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31, 2015.
[2] E. Amrani, Ben-Ari, D. Rotman, and Bronstein. Noise estimation using density estimation for self-supervised multimodal learning. In AAAI, 2021.
[3] X. Chen, H. Fang. T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L., Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv, 2015.
[4] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. UNITER: Universal image-text text representation learning. In ECCV, 2020.
[5] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and -at---nndd--ggggee tasks via text generation. arXiv, 2021.
[6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
[7] C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In CVPR, 2019.
[8] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. ArXiv, abs/ 16066068847 2016.
[9] J. Gao, R. Ge, K. Chen, and R. Nevatia. Motion cationapppearacce co-memory networks for video question answering. In CVPR, 2018.
[10] L. Gao, P. Zeng. Song, Y.-F. Li, Liu, T. Mei, and H. Shen. Structured two-stream attention network for video question answering. In AAAI, 2019.
[11] D. Huang, P. Chen, R. Zeng, Q. Du, M. Tan, and C. Gan. Location-aware graph convolutional networks for video question answering. In AAAI, 2020.
[12] Z. Huang, Z. Zeng, B. Liu, D. Fu, and J. Fu. Pixel-BERT: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004. 00849, 2020.
[13] Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim. TGIF-QA: Toward apatio-tempores reasoning in visual question answering. In CVPR, 2017.
[14] J. Jiang, Z. Chen, H. Lin, X. Zhao, and Y. Gao. Divide and conquer: Question-guided spatio- temporal contextual attention for video question answering. In AAAI, 2020.
[15] Jiang and Han. Reasoning with heterogeneous graph alignment for video question answering. In AAAI, 2020.
[16] W. Jin, Z. Zhao, M. Gu, J. Yu, J. Xiao, and Y. Zhuang. Multi-interaction network with object relation for video question answering. Proceedings of the 27th ACM International Conference on Multimedia, 2019.
[17] J. Kim, M. Ma. T. Pham, K. Kim, and C. D. Yoo. Modality shifting attention network for multi-modal video question answering. In CVPR, 2020.
[18] T. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ArXiv, abs/1 009.609999990 2017.
[19] R. Krishna, K. Hata, F. Ren, I. Fei-Fei, and J. C. Niebles. Dense-captioning events in videos. 2017.
[20] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 2017.

---

[21] M. Le, V. Le, S. Venkatesh, and T. Tran. Hierarchical conditional relation networks for video question answering. CVPR, 2020.
[22] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for video-and-languaar learning via sparse sampling. In CVPR, 2021.
[23] Lei, L. Yu, M. Bansal, and L.. Berg. TVQA: Localized, compositional video question answering, In EMNLP, 2018.
[24] L. Li, Y.-. Chen, Y. Cheng, Z. Gan, L. Yu, and J. Liu. HERO: Hierarchical encoder for Video+language omni-representation pre-training. In EMNLP, 2020.
[25] X. Li, J. Song, L. Gao, X. Liu, W. Huang, X. He, and C. Gan. Beyond RNNs: Positional self-attention with co-attention for video question answering. AAAI, 2019.
[26] X. Li, X. Yin, C. L.
X. Hu, P. Zhang. L. Zhang. L. Wang, H. Hu, L. Dong, F. Wei, Y. Choi, and J. Gao. Oscar: Object-semantics aligned pre-training for vision-languee tasks. In ECCV, 2020.
[27] Y. Liu, M. Out, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and
V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv, 2019.
[28] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[29] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and Sivic. H0wto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[30] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. Vetito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-perffrmance deep learning library. In NeurIPS, 2019.
[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. ArXiv, abs/2 103.00020, 2021.
[32] M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NIPS, 2015.
[33] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele. A dataset for movie description. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3202-3212. 2015.
[34] P. H. Seo, A. Nagrani, and C. Schmid. Look before you speak: Visually contextualized utterances. In CVPR, 2021.
[35] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. VideoBERT: A joint model for video and language representation learning. In ICCV, 2019.
[36] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP, 2019,
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. 2017.
[38] D. Xu, 7. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, 2017.
[39] C. Yan, Y. Tu, X. Wang, Y. Zhang, X. Hao, Y. Zhang, and Q. Dai. Stat: Spatial-temporal attention mechanism for video captioning. IEEE Transactions on Multimedia, 22:229-241 2020.
[40] T. Yang, Z. Zha, H. Xie, M. Wang, and H. Zhang. Question-aware tube-switch network for video question answering. Proceedings of the 27th ACM International Conference on Multimedia, 2019.
12

---

[41] Z. Yang, Z. Dai, Y. Yang. J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xinet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
[42] Z. Yang, N. Garefa, C. Chu, M. Otani, Nakashima, and H. Takemura. Bert representations for video question answering. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1554, 2020.
[43] Z. Yang, N. García, C. Chu, M. Otani, Y. Nakashima, and H. Takemura. A comparative study of language transformers for video question answering. Neurocomputingg 445:121-133, 2021.
[44] Yu, J. Li, Yu, and Q. Huang. Multimodal transformer with multi-view visual representation for image captioning. IEEE Transactions on Circuits and Systems for Video Technology, 30:4467-4480, 2020.
[45] W. Yu, J. Liang, I.. Ji, I.. Li, Fang, N. Xiao, and N. Duan. Hybrid reasoning network for video-based commonsense captioning. ArXiv, abs/2108.02365, 2021.
[46] W. Yu, J. Zhou, W. Yu, X. Liang, and N. Xiao. Heterogeneous graph learning for visual commonsense reasoning. In NeurlPS, 2019.
[47] Y. Yu, H. Ko, J. Choi, and G. Kim. End-to-end concept word detection for video captioning. retrieval, and question answering. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3261-3269, 2017.
[48] Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao. ActivityNet-QA: A dataset for understanding complex web videos via question answering. AAAI, 2019.
[49] 7. Zha, Liu, T. Yang, and Zhang. Spatiotemporal--exuaal co-attention network for video question answering. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 15:1 18, 2019.
[50] Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L.-J. Wang, Y. Choi, and J. Gao. Vinvl: Making visual representations matter in vision-language models. ArXiv, abs/2101.00529, 2021.
[51] L. Zhu and Y. Yang. ActBERT: Learning global-local video-text representations. In CVPR, 2020.
[52] Y. Zhuang, D. Xu, X. Yan, W. Cheng, Z. Zhao, S. Pu, and J. Xiao. Multichannel atten- tion refinement for video question answering. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 2020.
13


Reviews:
[{"review_id": "x-CECf5kXMO", "reviewer": "Reviewer_THcD", "summary": "The authors address the task of video question answering and extend the state-of-the-art framework using the concept of siamese sampling.\nThey build on the work of ClipBERT which proposed that using sparsely sampled frames from videos improve performance and efficiency compared to densely sampled frames.\nThis work performs multiple such sparse samples keeping temporal ordering (called as siamese samples) and combines prediction from the samples using graph convolutional layers leading to a pseudo label.\nThe final optimization function uses a combination of two loss functions: (i) loss between ground truth and predictions from anchor clip (a single sparse clip), and (ii) loss between predictions from anchor clip and pseudo labels generated from siamese samples.\nThey have shown that the model outperforms state-of-the-art significantly on several commonly used data sets.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. It's unclear why the authors used the predictions from siamese clips as pseudo labels instead of combining it with anchor clip predictions perhaps using the same graph convolutional architecture or other ensemble approach.\n2. The ablation studies section contains ablation with just Siamese Knowledge Reasoning (SKR) and both SKR and SKG. However, it's unclear if the gain in either of these cases is due to additional information from clips or just additional parameters and power of ensemble. The authors should perform ablation using multiple copies of anchor clip as input to SKG and understand the effect.\n3. The siamese sampling section (Section 3.2) is slightly unclear. Firstly, they mention they randomly select the starting sampling index of anchor clip. Do they have any constraints that it should be in first half of the video or based on the sampling stride the end index should be in range? Secondly, they mention the siamese samples occur \"nearby\" the index of anchor clip. Please define \"nearby\" concretely. Can the index be before the anchor clip? Is it consecutively after the anchor index?\n4. Notations in preliminary section (Section 3.1) do not specify the vector dimensions. Please specify them to increase clarity. Adding Algorithm can further improve readability.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations not discussed. The authors can do qualitative error analysis and downsides of using their approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "uStXPqr5U3S", "reviewer": "Reviewer_dFuU", "summary": "This submission addresses the problem of Video Question Answering (VideoQA). Compared to previous method, the authors propose to sample siamese samples from same video and apply siamese knowledge as a soft label to self-supervise train the network. Experiments show that the proposed method achieves state-of-the-art performance on five VideoQA datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n* The novelty is clear and motivation is reasonable. The technical details are explained clearly, with self-explained figures as figure 1 and figure 2\n\n* The SimSamRea method achieves state-of-the-art performances on five popular VideoQA datasets, which verify the generalizability and effectiveness of the method. \n\n* The siamese sampling and knowledge based self-supervised learning can be applied to future researches. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* It would be better to  release the reproducible code for future research purpose. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "fdyqU9CcXlg", "reviewer": "Reviewer_iH3u", "summary": "This paper tackles the video question answering task. It proposes a model with context reasoning within the same video, which is realized by generating pseudo-labels based on the visual similarity of sampled frames. Results are reported on five video question answering benchmarks, MSRVTT-QA, MSVD-QA, ActivityNet-QA, How2QA and TGIF-QA.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed model is technically sound. The paper is clearly written. The key components in the proposed model (e.g. the number of sampling siamese clips, siamese knowledge generation and siamese  knowledge reasoning) are ablated. Extensive evaluations are conducted on five benchmarks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The Equation (5) for generating the soft learning target Q contains trainable weights W1 and W2, what is the effect for these learnable weights? What if we directly use the rest part in Equation (5) (without learnable weights W1 and W2) as soft label?\n ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Q4oGYJ72p6", "reviewer": "Reviewer_JZcw", "summary": "This paper proposes a self-driven siamese sampling and reason (SiaSamRea) method for the video question answer task. The task is essentially solved by a classification or regression model based on the problem settings. The key idea of SiaSamRea is to sample more clips in the training phase so that these extra clips can provide soft labels, which are similar to pseudo labels, for another supervision. The Siamese Knowledge Generation (SKG) module builds an interdependent matrix to measure the correlation between each clip-pair of the video, and the Siamese Knowledge Reasoning (SKR) module aggregates the prediction from these extra clips based on the correlation matrix from SKG. Finally, the aggregation in SKR is used as a soft label and included in the loss term. By doing so, the method achieves state-of-the-art performance on five VideoQA benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) The idea of this paper is intuitive and easy to follow. This is also because of the clear writing and illustrative features. \n\n(+) The SKR module seems to be very effective according to the ablation study. This might be due to the soft label is usually useful to optimize the model. The supervision from Siamese Clips's prediction provides more in-the-domain information to the model.\n\n(+) This work reaches state-of-the-art performance on five different datasets. If the code gets released as the authors promised in the paper, it will be a good resource for the video question answering community.\n\n(-) The SiaSamRea method is a bit mismatch to the video question answering task. This method should be aligned to a more generic video understanding task such as action classification, or even self-supervised learning such as SimCLR, SwAV, and SimSam, because the proposed SKG and SKR modules only require the input should have video. \n\n(-) Although it shows superior performance on five different datasets, the comparisons are not very fair. As we know, soft labeling is a strong method to improve model training. According to the ablation experiments, we can also see that only use SKR can give a big boost (~4%). However, the author is comparing their methods with other non-soft-labeling ones. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would like to suggest comparing the effect of soft labeling with the proposed SKG and SKR modules. Soft labeling, or generating pseudo labels from the training set, usually gives an extra boost to the model performance. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed recommendations from the reviewers, one reviewer increasing their score after the author response leading to 2x5, 1x6, and 1x7 rating.

One concern remaining after the author response was the question if the novel approach is specific enough to VideoQA or should be evaluated on other video recognition tasks.
However, I think
(a) answering this question is outside the scope of this paper, and remains to be seen, as the question/task conditioning might be an important component which makes the proposed approach so successful and if future work shows this is generalizable to other tasks will only increase the impact of this work further.
(b) the strong results (in combination with the novel approach in the videoQA field) are of interest to the community

Given the interesting model and strong results (outperforming recent work, including CoMVT [34] from CVPR 21) on 5 datasets, and solid ablation study, I recommend acceptance under the expectation that the authors revise the paper according to the author response and address the reviewers concerns, including, but not limited to:
1) ablating W1, W2
2) ablation with multiple copies
3) release code
4) additional details about where the sampling of the anchor and Siamese clips is 

Additional I would like the authors to add the missing comparison to [22] on TGIF and recommend proofreading the paper for English fluency.
Decision: Accept (Poster)