Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Authors: Weijiang Yu
Haoteng Zheng
Mengfei Li
Lei Ji
Lijun Wu
Nong Xiao
Nan Duan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video Question Answering
Multi-view Learning
Data-effeciency Reasoning

Abstract: Recent advances in the video question answering (i.e., VideoQA) task have achieved strong success by following the paradigm of fine-tuning each clip-text pair independently on the pretrained transformer-based model via supervised learning. Intuitively, multiple samples (i.e., clips) should be interdependent to capture similar visual and key semantic information in the same video. To consider the interdependent knowledge between contextual clips into the network inference, we propose a Siamese Sampling and Reasoning (SiaSamRea) approach, which consists of a siamese sampling mechanism to generate sparse and similar clips (i.e., siamese clips) from the same video, and a novel reasoning strategy for integrating the interdependent knowledge between contextual clips into the network. The reasoning strategy contains two modules: (1) siamese knowledge generation to learn the inter-relationship among clips; (2) siamese knowledge reasoning to produce the refined soft label by propagating the weights of inter-relationship to the predicted candidates of all clips. Finally, our SiaSamRea can endow the current multimodal reasoning paradigm with the ability of learning from inside via the guidance of soft labels. Extensive experiments demonstrate our SiaSamRea achieves state-of-the-art performance on five VideoQA benchmarks, e.g., a significant +2.1% gain on MSRVTT-QA, +2.9% on MSVD-QA, +1.0% on ActivityNet-QA, +1.8% on How2QA and +4.3% (action) on TGIF-QA.

Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Weijiang Yu', Haoteng Zheng', Mengfei Li', Lei Ji2, Lijun Wu?, Nong Xiao Nan Duan? School of Computer Science and Engineering, Sun Yat-sen University
Microsoff Reasearch Asia
wei aiigiangyu&@gmal com, xiaon momom60mail.. ees.mmmlll...eeee...e edu.
(zhenght5, 1imf33/ @mmai12 sysu edu.
(leiji, Lijun.wu. wu, nanduan @microsoft. com
Abstract
Recent advances in the video question answering (i.e., VideoQA) task have achieved strong success by following the paradigm of fine-tuning each clip-text pair independently on the pretrained transformer-bes based model via supervised learning. Intuitively, multiple samples (i.c., clips) should be interdependent to capture similar visual and key semantic information in the same video. To consider the interdepen- dent knowledge between contextual clips into the network inference, we propose Siamese Sampling and Reasoning (SiaSamRea approach, which consists of siamese sampling mechanism generate sparse and similar clips (i.e., siamese clips) from the same video, and novel reasoning strategy for integrating the inter- dependent knowledge between contextual clips into the network. The reasoning strategy contains two modules: (1) siamese knowledge generation to learn the inter-relationship among clips; (2) siamese knowledge reasoning to produce the refined soft label by propagating the weights of nnter-relationship to the predicted candidates of all clips. Finally, SiaSamRea can endow the current multimodal reasoning paradigm with the ability of learning from inside via the guidance soft labels. Extensive experiments demonstrate our SiaSamRea achieves state-of- the-art performance on five VideoQA benchmarks, e.g., significant +2.1 % gain MSRVTT-QA. +2.9% MSVD-QA, +1.0% ActivityNet-QA, +1.8 % on How2QA and +4.3 % (action) on TGIF-QA.
Introduction
By inferring the correct answers for video-based questions, video question answering (VideoQA) has attracted increasing research attention due its huge application potential, as fundamen- reasoning.
Dense Clip
Sparse Clip
Siamese Clips
tal technique for vision-to-langen The task involves acquisition and manipulation of spatio-temporal visual representations guided by the compositional semanties of the linguistic clues [32 [15
[a] Dense Sampling (b) Sparse Sampling
(c) Siamese Sampling
Figure 1: Different sampling mechanisms for video frames. (a) Traditional methods use dense clip features from full-length videos. (b) A re- cent approach [22 suggests sparsely sampled clips for end-to-end learning. (c) Our siamese sampling to generate similar semantic clips.
21 34]. Existing works can roughly be divided into two aspects. One aspect to ex- plore powerful multimodal transformer-basee net- work [22 2 34 45 trained on large-scale datasets
(e.g., COCO Captions
3 and HowTo100M, (29]).
*Corresponding Author: xiaon6 @ www.aaai......u..mmmmm
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

knowledge
Guideraa
Beasnning
Panchor
[p,, Pa Pal
Prediction
Prediction
Clessifler
Classifier
Stamesse
Know wage
Classifier
Classifier
f anchor
f
f
Cross-modal
Cross-modal Model
Cross-modal
Cross-modal Model
Model
Stop Gradient
t
V
V anchar
Encoder
Video
Text Encoder
Video Encoder
Text Encoder
Video Encoder
Encoder
Video
Text Encoder
Sampling
Frame
Word Embedding
Sparse Frame Sempling
Embadding
Sampling
Sampling
Stameee
Embedding
Word
Video
Text
Video
Video
Text
(x) Traditional Multimodal Learning
(b/ Sparse based Multimadal Learnnng
(c) Self-drivo driven based Multimodal Learning
Figure 2: (a) Traditional multimodal learning uses dense sampling for video frames and extracts individual clip-text features via offiin encoders. (b) Sparse based multimodal learning benefits from modeling. (c)
sparsely sampled clips (independent) and raw text tokens for based multimodal learning utilizes cross-relationship between anchor clip (Vanchor) and siamese clips (V1, V2, Va) as knowledge lead the model to create soft labels for self-supervised learning. Note that the modules in (c) using same color mean the weights are shared.
The other aspect aims at exploring the structure reasoning for semantic alignment between vision and language (c.g.. Hicrarchical Reasoning [21]. Heterogeneous Graph Aligmment [155 46 and Object Relation Reasoning [6].. Both of them solely consider each clip-text pair separately and ignore the correlation between contextual clips in the same video.
By further analyzing the existing multimodal learning paradigm and showing their difference in Figure [ we observe that current methods suffer from key drawback: every clip-text pair is regarded individual and independent during the training. Such drawback overlooks the rich interaction of contextual clips from the same video? We believe that the internal interaction information from same videos can be helpful for further enhancing the network learning. Hence, provide remedy to this dilemma, we present self-driven Siamese Sampling and Reasoning (SiaSamRea) framework learning from inside, by using the internal contextual semantics of interdependent video-aware data (e.g., clips) from the same video in the training process.
First of all, SiaSamRea (in Figure consists of two key parts: (1) siamese sampling shown in Figure (c) to extract multiple similar clips from the same video, which is motivated by ClipBert [22]; (2) reasoning strategy named self- based multimodal learning as shown in Figure 2 (c). The strategy contains two modules: (i) a siamese knowledge generation module to calculate the correlationn matrix between the anchor clip-text pair (e.g., fanchor) and siamese clip-text pairs (e.g., fū, f2, fa); (61) a siamese knowledge reasoning module to produce soft labels for self-supervised refinement during the training. Finally, the labels are applied as auxiliary training supervision to enhance the network.
Compared with previous sampling mechanism, our siamese sampling as shown in Figure D (c) not only is sparse but also is able generate multiple similar clips for constructing their internal relationships. Specifically, our siamese sampling captures clips at different start frames with same interval time in the same video, which constrain the global semantic of each clip to be similar (i.e., each clip can represent the consistent video content from global perspective).
Furthermore, to fully utilize the siamese clips, we explore a new reasoning strategy namely self-driven based multimodal learning as shown Figure (e). There are three steps in the strategy for using the interaction of internal clips from the same vidco into the training. At the 1s step, the anchor clip and siamese clips are obtained by using sparse sampling and siamese sampling, respectively. Then the two types of clips individually cooperated with the text are fed into the model to extract clip-text features, including anchor clip-text feature (e.g., Vanchor ) and siamese clip-text features (e.g., V1, V2, V3). At the 2nd step, the internal contextual interaction between the anchor clip-text feature and siamese
2 We call the contextual clips from same video as internal clips containing anchor and siamese clips The interal contextual interaction is regarded as siamese knowledge in our paper.

---

clip-text features is calculated via our siamese knowledge generation module. At the 3rd step, siamese knowledge reasoning module is proposed use the siamese knowledge applying to several predicted candidates (e.g., Panchor; PI, Pt P2, Pa) for adaptively reasoning out the refined soft label. Here, the siamese knowledge is applied for adaptively reasoning. Because is hard to distinguish which candidate is critical to the accuracy from several predictions during the training. Finally, we use the refined soft label to further distill our model for high-quality representation generation.
Our contributions are three-fold: (i) We propose novel end-to-end framework named SiaSamRea for learning from inside on VideoQA task, by using siamese sampling and reasoning to integrate the interdependent semantics of clips from the same video into the training process. (ti) A novel reasoning strategy is carefully designed for building the soft guidance from the interdependent knowledge between internal clips, which consists of a siamese knowledge generation module and siamese knowledge reasoning module. (iii) Experiments on five commonly-used VideoQA benchmarks show the superior ability of our SiaSamRea and demonstrate the effectiveness of our proposed components. Not that our method only teaches the network with interdependent knowledge during the training. which does not bring any extra burden (e.g., computation, memory and parameters) in the inference.
2 Related Work
Visual Content Modeling on VideoQA. Video Question Answering is a task aiming to answer the given question concerning video content. Some current works extract generic visual appearance and motion features to represent video contents and design different attention mechanisms to integrate these features, like question-routed attention 38 14
and co-attention p These methods mainly focus on the holistic understanding of video contents, which easily neglect the meaningful details of local clips. Some research pays attention to structure reasoning for semantic alignment [6]] [I]
[7]. Jiang et al. [15) proposed to build the correlation between the inter- and aitra-modalities for cross-modal learning via a heterogeneous graph alignment framework. Huang et al. [ presented novel location-awe aware graph convolution network to mine the structural representation between the location and relation among visual objects. Le et al. [2T] demonstrated the effectiveness of hierarchical video features for VideoQA by developing a general- purpose neural reasoning unit. These works aim to handle fine-grained visual entity (e.g., object) and multimodal hierarchy but ignore the correlation between contextual clips in the same video.
dhetrained-innttuning Paradigm on VideoQA. Recently, the paradigm of pretrained fine-tuning has made significant progress in many multimodal tasks, such as text-video retrieval 33 visual question answering M. image captioning
44 video captioning 39] and video question answer-
ing [23 Thanks to the strong success of transformer-
[37 language pretraining 6 27
35| has shown promising improvement on the video- langu-ang tasks, especially on VideoQA task. For example, Zhang et al. (50) proposed to enhance the visual representation based on the OSCAR (26 by in- corporating more visual data during the training. Lei et al. [22 solved the offline encoder in the netrained-finetuninn pipeline by adopting a sparse training and dense inference. To promote the development of large-scale multimodal learning, Seo et al. (34] proposed a new dataset and task for future utterance prediction based on the video-text inputs. The multimodal model trained on their new dataset also performed well on the VideoQA task. It can further prove the transfer ability of the pretrained- pretrained-finetuinng paradigm. All approaches as mentioned-above on VideoQA are benefited from the powerful pretrained backbone with large-scale clip-text pairs, while each clip-text pair is independently encoded into the network rather than mining their contextual knowledge in the same video. In this paper, we argue that the rich contextual information among clips should be valuable and can well enhance the network. Hence, we propose a siamese sampling and reasoning method for learning from internal clips in the same video as well as bringing additional burden in the inference, which consists of a siamese sampling mechanism and a reasoning strategy.
and image-text pretraining [36 5 B1) the video-text pretraining [24 29 51
3 Methodology
Our Siamese Sampling and Reasoning (SiaSamRea) method shown in Figure The SiaSamRea can be roughly divided into three parts based on the learning pipeline: (1) Clip Sampling; (2) Feature Extraction: (3) Reasoning Strategy. The clip sampling aims to get the anchor clip and siamese clips by sparse sampling and siamese sampling, respectively. The feature extraction is composed of a video
3

---

Video Frames
Anchor Clip
Video
V
Multimodal
FC
Classifier
Transformer
Guidance
Fircher
Punchr
Share Weights
Share Weights
Text
Stamese
Siamese
Encoder
Knowledge
Knowledge
Generation
Reasoning
Slamese Clips
Textual Inputs
fufufs.
PuPpPPs
Video
N,N,
Multimodal Transformer
Encoder
FC
FC
Classifier
Siamese Sampling
Reasoning Strategy
Figure 3: Overview of our Siamese Sampling and Reasoning (SiaSamRea) applied to VideoQA. Given the dense video frames, the anchor clip and siamese clips first to be extracted by sparse sampling and siamese sampling. Then a video/text encoder and a multimodal transformer are utilized to generate multimodal features (fanchor: f1, f2, fz). Next, siamese knowledge generation module is proposed to produce contextual relationship between anchor clip and siamese clips from the same video. Finally, the soft label used for distilling the whole network, can be obtained via siamese knowledge reasoning module by inferring the siamese knowledge on several candidates (Panchor P1, P2, pa). Note that we only need the network fed with anchor clip for inference.
encoder, a text encoder and a multimodal transformer, which encodes the multiple clip-text pairs as semantic representations. The goal of the reasoning strategy is generate refined labels for distilling the model learning, which can be achieved by using the siamese knowledge reasoning module to propagate the siamese knowledge (i.e., interdependent relationship among pairs) from the siamese knowledge generation module into the predictions of multiple clip-text pairs. Our contribution mainly single framework, the siamese sampling mechanism
focuses on the design of reasoning strategy and innovative modules (e.g., siamese knowledge generation and siamese knowledge reasoning), which are unveiled and discussed in details in the following sub-sections.
3.1 Preliminary
To simplify our method, we unify some symbolic notations in this section. Given the dense video frames F, we seek to construct the anchor clip Canchor and siamese clip set Csiamese = [c N-1 where the N means all clips number. Each clip is uniformly sampled frames and obtains B feature maps by using the video encoder to encode the clip. The encoded features of anchor clip and the siamese clip set are denoted as Vanchor and yssiames = [ N-1
correspondingly. Taking the encoded text representation from the text encoder concatenated with the clip representation from the video encoder as input, multimodal transformer is applied to produce the clip-text feature. Each anchor elip-text feature is indicated fanchor and the associated siamese clip-text feature set indicates Eviamese
Ji=1
{ Similarly, their predictions are defined as Panchor and Psiamese WP respectively. The is a vector with K dimension according to the classification number.
3.2 Siamese Sampling
In this section, we introduce our siamese sampling shown in Figure (i), which is inspired by the spare sampling in ClipBert (22). First, we get an anchor clip Canchor by sparsely and uniformly sampling B frames from dense video frames F. The length of typically much larger than B. We randomly decide the starting sampling index of the anchor clip. Then our siamese sampling captures siamese clips different starting indexes but nearby the index the anchor clip in the same video. We use the same interval frame for all clips. There are some constraints for selecting the starting sampling index of the anchor clip. should be the first third of the video to guarantee equilibrium of sampling. The "nearby" means the index of the Siamese clip should be next to the index of the anchor clip. The pace of sampling is consecutive before or after the anchor index. After obtain some siamese clips, which have similar global vidco semantics
siamese sampling,

---

including the anchor clip. Then we feed these clips into the video encoder sequentially as shown in Figure to get the visual feature. The video encoder and text encoder are similar with the encoders in the ClipBert [22].
3.3 Reasoning Strategy
that directly utilized the clip-text features from multimodal transformer to predict the final results, we propose a novel reasoning strategy named self-driven multimodal learning by exploring how integrate knowledge between different clips (i.e., siamese clips) in the same video with single network, which consists of a siamese knowledge generation module and siamese knowledge reasoning module.
Different from previous methods 442 22
3.3.1 Siamese Knowledge Generation
Intuitively, clips with high visual similarities expected to have more consistent predictions on their predicted class probabilities, regardless their ground-truth labels. In our solution, similar clips' knowledge is systematically aggregated and integrated provide better soft labels.
We propose to propagate and integrate knowledge among clip-text pairs on-the-fly in terms of their multimodal feature similarities. Given set of N clip-text pairs from the same video and feature extractor F under training, we first estimate the samples' pairwise similarities by the dot product of their encoded representations with the current network. Such similarities can be stored in an interdependent matrix A E IR N as
Ai.j) (FF()) (FFE)))))))(((((())
where j the indices of samples in video and denotes the 82-norm function. N is the number of samples. To avoid reasoning in the self-loop reinforcement, we discard the diagonal entries from A by A A (1 I). where I is an identity matrix and @ denotes the Hadamard product. Subsequently, we normalize each row of the interdependent matrix so that Σ Aii,j = 1 for
all i, while keeping the diagonal all zeros, i.e., A i) O. The normalization can be formulated as softmax function over each row of the matrix A
(x,(A(i,j) A(i Ejtt exp A (i.j))
Vi NJ.
where the jii indicates the normalized interdependent knowledge between the i-th sample and j-th sample in the same video. The whole interdependent knowledge of all sampling clips denoted as À is also called as siamese knowledge in this paper. Due to the knowledge mainly calculated by our siamese samples.
3.3.2 Siamese Knowledge Reasoning
We denote the predicted probabilities of samples within a video as P = .p......pN] E RNK, which satisfy ΣK ((i, k) 1, Vi. For the i-th sample in the video, there are K-class candidates. We would like to adaptively propagate and merge the other samples' predictions to create a better soft label for it based on the inter-sample affinities, which can be formulated as
p. A( - (())P
(3)
where P. is the propagated probability vector for the i-th sample and can serve as the refined soft labels. Intuitively, if the i-th sample and the j-th sample are similar with a high interdependent value (i..), the prediction P, would have a larger weight to be propagated to p;- Motivated by the graph convolutional network (18), module adaptively propagate the predictions between all the samples (i.e., clip set) in a video in parallel, which can be formulated as
P WAP,
(4)
where the W E RIXN is learnable matrix and P E R is our soft label. To avoid propagating and integrating noisy and unexpected predictions too much, we produce the soft learning targets Q as
"o simplify the expression, the "clip-text pair" denotes "sample" in Sec. 3.3
5

---

weighted sum of the initial probability matrix P and the propagated one AP,
WW (1-
(5) k=1 Q(k) - = 1, R The W1 and W2 are
where E [0. 1] is the weighting factor and Σ Σx trainable weights. With the above formulations, the knowledge between samples within the same video can be propagated to each other and integrated for one iteration.
3.4 Optimization
Open-ended VideoQA Setting. Open-ended question setting is to choose one correct answer from a predefined answer set Ω which can be seen as a multi-label classification task and trained with the cross-entropy loss function. We feed the visual representation and question representation q into our framework denoted as dg to compute prediction probabilities:
p (v q). R!
Multiple-choice VideoQA Setting, Multiple-choice question setting is to choose one correct answer from M candidates. In this case, first formulate the answer representation of the m-th candidate as am- Then, we feed the visual representation, question representation and answer representation into our model ouput the m-th answer score, which formulated as
8m (v q am), 1 S S M,
where the score of the correct candidate is the positive score s", and the rest scores are negative scores denoted as (sf, ..., 2)). During training, we utilize the summed pairwise hinge loss
1= maz(0, 1 (s" s))) between the positive score and each negative score to train our model.
M-1
terms for our final objective function L. One term denoted as Sisemese comes from the loss between our soft label and the prediction result Panchor from the anchor clip-text pair. We utilize the cross-entropy loss function to compute the loss Lisamesee- The other term is to optimize the training process between predictions and the ground-truth, which is commonly in the previous works. Note that the last term may be different depended the task setting, we uniformly formulate the loss function associated with the ground-truth as Cgz. Finally, our model can be finally optimized by
Objective Function. There
-= L= + Leghs
(8)
where a is hyper-paramest term to adjust the balance of the two losses.
4
Experiment
In this section, we first introduce the data that we use in Sec. 4.1 and details of our experiments in Sec. 4.2 Then we validate the effectiveness of our proposed components in Sec. 4.3 which is
followed by the comparison with other methods in Sec. [4.4, Finally, in Sec. 4.5) some visualization
results are shown to qualitatively analyze the benefits of our siamese clips.
4.1 Evaluation and Datasets
We evaluate our proposed VideoQA method and compare it with other state-of-the-a methods on five VideoQA datasets, which are widely used and accepted for academic video question answering, including the open-ended setting and multiple-choice setting. We follow the previous evaluation protocols for open-ended settings [21 (48) and utilize a fixed vocabulary training answers. For the multiple-choice evaluation setting, we follow the previous methods [21] to use the same dataset split setting. Excepting for the repetition count task in the TGIF-QA [3] where the Mean Squre Error (MSE) is excavated, we use top-1 accuracy to be the evaluation metric for all experiments.
We carry out our SiaSamRea on three open-ended VideoQA datasets, including MSRVTT-QA (8), MSVD-QA [8 and ActivityNet-ON QA [48]. The MSRVTT-QA has 10K videos and 244K question- answer pairs. The size of the predefined answer set 1000. Compared with the MSRVTT-QA. the MSVD-QA is smaller, which has 2K videos with nearly 51K question-answer pairs. The MSRVTT- QA and MSVD-QA contain five question types like what, who, how, when and where. ActivityNet- QA has 5.8K videos and 58K questions, which is repurposed from ActivityNet Captions [1]]

---

Furthermore, we validate our method on a multiple-choice VideoQA datasets, such as How2QA [24]. The How2QA has 9K videos tailed with 44K question numbers. where each question associated with one correct and three incorrect answers. Besides, we train and test our model on the TGIF-QA [3] which is a large-scale VideoQA dataset containing 72K animated GIFs and 165K question-answer pairs. There are four tasks defined on this dataset: (x) Repetition Count is an open-ended QA task to
count the number of the repetitions of an action; (ti) Repeating Action is similar to the repetition
count task but following the multiple-choice setting: (iii) State Transition is multiple choice task for identifying the transition between two states; (iv) Frame QA is open-ended to find the sufficient frame to answer the questions.
4.2 Implementation Details
We implement our proposed SiaSamRea via PyTorch [0), a well-known open- open-source deep learning framework. For being fairly comparable with other methods, we conduct the following training and testing settings. Unless otherwise noted, settings are the same for all experiments.
Training. We use the pretrained model trained on the COCO Captions [3] and Visual Genome Captions [20] from ClipBert [22] as our initial weights of our SiaSamRea. To obtain more Vi related pretrained weights, we present to train task-related data during the pretraining. Based on the initial weights from the ClipBert, we perform the clip-text pretraining on TGIF-QA [13] optimized by adaptively using aforementioned objection function (i.e., Eq. 8) according to different task settings (i.e.. open-ended and multiple-choiee tasks). There a total of 71739 videos in the TGIF-QA dataset. Because our proposed method aims to model the long-range contextualized knowledge via Siamese samples. The videos with less than 40 frames do not meet our requirements in the training. We filter out less than 40 frames of video during the pre-training to remain long-range temporal videos. In the TGIF-QA QA dataset, the percentage of videos with less than 40 frames 56% (40333/71739). Hence, we use the remaining 3 31006 TGIF videos for our pre-training, which is less than the ClipBERT. The ClipBERT uses a total of 71739 TGIF-QA videos to finetune the four tasks, which different from our pre-training on the TGIF-QA dataset. Because the video length is various, we set the number of frames in each clip from to 16 according to the video length. We set the batch size to per GPU and use AdamW (28) optimizer with initial learning rate 0.0005. The learning rate warm-up strategy is adopted over the first 10% training steps followed by linear decay to zero. We terminate the pretraining when it reaches minimal learning rate (i.c., 0.00001) and its validation performance stabilizes (i.e., reaches convergence). The balance term is o=1.0, the integrating weight is w=0.5. We conduct all experiments using 16 NVIDIA V100 GPU cards.
For the downstream fine-tuning, we apply the same training and optimizer configurations as mentioned-above. Since the downstream datasets are various and in multiple domains, we use dataset-specific learning rates, training epochs and clip numbers based on validation performance. Note that we do not need to excavate additional fine-tuning on the TGIF-QA dataset.
Testing, We strictly follow the test split setup in each dataset and validate the learning outcomes after each learning epoch. Then we report the best metrics (top-1 accuracy) as our results.
4.3 Ablation Studies
We conduct comprehensive ablation studies con- cerning various aspects of our SiaSamRea's 's de- sign in this section. In Table [ we set the network training without using the Lviamese as baseline. Compared with the baseline, our method only using the siamese knowledge reasoning module (w/ SKR) by replacing the siamese knowledge generation with average op- eration, can apparently boost the accuracy by more than 4% from baseline on How2QA and MSVD-QA. can demonstrate the effectiveness of our siamese knowledge reasoning. It also validates our assumption that multiple clips from same videos can really bring some strong knowledge to en- hance the network ability. When we add the siamese knowledge generation module (w/ SKG+SKR), our method can reach a higher performance to 84. 1 1% and 45.5%, which can demonstrate the ef- fectiveness of building the interdependent knowledge (i.e., siamese knowledge) for integrating the samples. It can also prove the adaptively reasoning by our siamese knowledge is better than the purely
Table 1: Ablation studies on How2QA and MSVD- QA datasets. The SKG and SKR separately indi- cate siamese knowledge generation and siamese knowledge reasoning.
Methods baseline
How2QA
MSVD-QA
79.1
39.4
w/ SKR
83.0
44.7
w/ SKG SKR
84.1
45.5
7

---

average operation. We think that the siamese knowledge not only serves as the knowledge-rouse routed representation, but also implicitly constrains the semantic consistency of clips in the space of clip-text features.
To explore the effect of the number of siamese clips, we compare our method with various number of siamese clips, which results can be shown in Table 2 Note that our siamese sam- ples are serving for the interdependent knowl- edge extraction. Intuitively, more samples can aggregate richer knowledge. When adding the number of siamese samples from 1 to 12, our method promotes the accuracy from 79.6% to 84.4% on How2QA and 39.9 59.9%-445.7 on MSVD-QA. Such improvement can demonstrate our intuition that more siamese samples can bring richer knowledge, which makes our network benefited from it. Although the accuracy increasing with the number of siamese clips, we observe that the improve- ment from the number 8 to 12 is slight. We think the reason is the saturation of knowledge, i.e., the model has enough knowledge to learn the task on the two datasets. Hence, it is almost meaningless to purely increase the siamese samples. To balance the training time and accuracy, we select the 8 siamese samples in our final version.
Table 2:
The effect of the number of sampling siamese clips on How2QA and MSVD-QA.
Methods How2QA MSVD-QA
2
8
12
79.6
80.5
81.9
84.1
84.4
39.9
41.3
42.7
45.5
45.7
The effect for WI and W2 in Eq. 3 We have conducted experiments on the MSVD-QA dataset to analyze the effect of these learnable weights (WI and W2). Our SiaSamRea without WI performs 44.9% on MSVD-QA, which heavily drops 0.6% performance compared with our whole SiaSamRea (45.5%). One possible reason that the W1 can make the propagated prediction matrix learnable and adaptively propagate the predictions between all the samples. When performing our model without W2, it achieves 45.2 5..2%. The performance without W2 is slightly decreasing ( 0.3%). When comes to method without using WI and W2, it gets 44.5%. As we can see, the WI and W2 both can bright improvement. Hence, it can further demonstrate the importance and necessity for the generation of expected soft labels by correctly merging several predicted candidates.
Table 3: Comparison with state of the art on MSRVTT-QA and MSVD-QA (top-1 accuracy).
The effect for multiple copies of anchor clip. We replace our Siamese clips with multiple copies of anchor clips to analyze the effect of SKG. Since we used 8 siamese clips in our paper, we replaced the siamese clips with 8 copies of anchor clips during the training. We execute this ablation study on the MSVD-QA dataset. This ablation study achieves 41.3%. It promotes the baseline (39.4%) with 1.9% performance. It can validate the power of the ensemble way. But only using the anchor clips for ensemble can't bring visually contextual- ized knowledge. We skillfully exploit the visually contextualized knowledge via the siamese clips, which obtains 45.5% accuracy. Hence, the im- provement of our method mainly comes from the SKR and SKG by using Siamese clips rather than the simple ensemble.
Methods
MSRVTT-QA MSVD-QA
E-SA [38]
29.3
27.6
ST-TP 13
30.9
31.3
AMU 38
32.5
32.0
Co-mem 9
32.0
31.7
HME 7
33.0
33.7
LAGCN
34.3
HGA [15
35.5
34.7
QueST 14
34.6
36.1
MINOR [16
35.4
35.0
TSN [40]
35,4
36.7
HCRN 21
35.6
36.1
Clip-BERT 22
37,4
SSML
35.1
35.1
CoMVT
39.5
42.6
SiaSamRea (Ours)
41.6
45.5
4.4
Comparisons with State-of-the-Arts
Comparisons on MSRYTT-QA and MSVD- QA. Table reports the comparison with previous methods on MSRVTT-QA and MSVD-QA datasets. Our SiaSamRea achieves the best performance on both datasets. Specifically, compared with the methods using large-scale pretrained data (i.c., Clip-BERT, SSML and CoMVT), our method consistently and considerably outperforms them by 2.1% and 2.9% top-1 accuracy on MSRVTT-QA and MSVD-QA when compared with CoMVT, which was previously the best performer. When comes to other methods (e.g., HCRN, TSN etc.) without using additional pretrained data, our model can significantly boost the accuracy by around 6% accuracy on MSRVTT-QA and a large gain of 9.4% on MSVD-QA compared with HCRN. These promising results can validate the effectiveness of using pretrained data, It also supports the feasibility of distilling the interdependence of multiple clips with our reasoning strategy on the open-ended VidcoQA task.
8

---

our SiaSamRea consistently out- performs state-of-the-arr models Comparisons on ActivityNNet-QA and How2QA. We now move to the evaluation on ActivityNet- QA and How2QA, which results are reported in Table 4 In particular, our method improves over the recent CoMVT approach that has been pretrained on HowTo100M [29] dataset. It can effectively demonstrate the advantages of apply- ing related-data (c.g.. TGIF-QA) on the pretrain- ing, which can perform well to transfer to related tasks. The ActivityNet-QA is so challenging due to the complex events and backgrounds in the videos. Our SiaSamRea still achieves a new state- of-the-art performance (39.8%) on ActivityNet- QA by learning from internal clips. These strong results show the importance and powerful ability of our proposed learning strategy. In other words, SeaSamRea can provide another way handle the complex situation by mining the intrinsic information like internal relationship to parse the various temporal scenes from multiple samples. On the multiple-choice task, the ability of our method gets great scores (84.1%) 1%) on How2QA dataset as validating the availability of transferring our method to the other video question answering settings.
Table 4: Comparison with state of the art on ActivityNer-QA and the public val set of How2QA (top-1 accuracy).
Methods
ActivityNet-QA How2QA
E-SA 48
31.8
MAR- VQA
34.6
HERO 24
74.1
CoMVT 33
38.8
82.3
SiaSamRea (Ours)
39.8
84.1
Table 5: Comparison with the state-of-the-ar methods on TGIF-
Comparisons TGIF-QA. To further demonstrate the advantage of SiaSamRea to infer more vari- ous and difficult scenarios, we es- timate it the TGIF-QA dataset. The experimental results are sum- marized in Table 5 It is clear that
QA dataset. For count, the lower the better.
Methods
Action
Trans.
Frame
Count
VIS+LSTM (agg)
32
46.8
56.9
34,6
5.09
VIS+LSTM (avg)
32
48.8
34.8
35.0
4.80
VQA-MCB (agg)
8
58.9
24.3
25.7
5.17
VQA-MCB (avg)
29.1
33.0
15.5
5.54
CT-SAN 47
56.1
64.0
39.6
5.13
ST-TP 13
62.9
69.4
49.5
4.32
on four reasoning tasks. The four tasks are requiring the strong tem- poral reasoning especially for the action and transition task. Hence,
GR-ATT 38
68.8
73.9
53.0
4.32
Co-mem
68.2
74.3
51.5
4.10
PSAC 255
70.4
76.9
55.7
4.27
STA I0
72.3
79.0
56.6
4.25
they really require the model to consider the contextual depen- dence of clips. Particularly, our method increases overall accuracy on the publicly test set by 4.3% 75.4%~779.79 compared with the best result on repeating ac- tion task, 3.9% (81.4%++85335)
MiNOR 16
72.7
80.9
57.1
4.17
HME 7
73.9
77.8
53.8
4.02
HCRN 21
75.0
81.4
55.9
3.82
ClipBERT
75.1
80.8
56.3
4.07
HGA [1
75.4
81.0
55.1
4.09
SiaSamRea (Ours)
79.7
85.3
60.2
3.61
FrameQA task and decreases error by 0.21 (3.82+3.61) on repetition count task. Thanks to the siamese sampling and reasoning, our method effectively capture the consistent video content from a global perspective. Because SiaSam- Rea can distinguish the global semantic of a video by multiple views (i.c., siamese clips). which can bring abundant information from inside that may be absent in other clips to help the network learning the contextual relationship.
state transition task, 3.1% (57.1%-660.25%
To fairly compare the effect of ClipBERT with proposed method, we conduct extra experiments to finetune the CRT on the filtered data and test it on the four tasks. We mark the ClipBERT finetuned our filtered data as ClipBERT* The ClipBERT* gets 75.1 % on Action, 80.8% on Trans., 56.3% on Frame and 4.07 on Count. As we can see, the ClipBERT* can achieve comparable results compared with other SOTA methods. It can prove the effectiveness of the pre- treitriined weights from COCO Captions and Visual Genome Captions. However, our proposed method still outperforms the ClipBERT* on four tasks based on the same filtered data (79.7% on Action, 85.3% Trans., 60.2% on Frame and 3.61 on Count). It can further demonstrate the superior ability of our idea by using the Siamese clips during the training. Our proposed method achieves better performance with less data. also validates the huge potential of inside data once there a suitable way to explore them.

---

Q: What perched on a branch of tree bit a tail of another animal? A: Bird
Anchor
Clip
Siamese Clips
Figure 4: The examples to show the benefits from the siamese clips. It is hard to discriminate the visual content "bird" from ambiguous parts (red circle) in anchor clip. It much easier distinguish "bird" by the assistance of siamese clips that contains more complete visual content (green circle).
4.5 Visualization
In Figure 4 we show training example to qualitatively analyze what kind of knowledge does the siamese clips bring to the anchor clip. is unavoidable to lose some visual contents when sparsely sampling from the video. For example, the visual content "bird" in anchor clip is almost absent. However, we can obtain the absent content from the siamemse clips due different sampling indices and duration. Hence, our siamese samples can help the model to correctly and stably train on the sparse samples as well as avoiding confusing predictions due to the incomplete visual information.
5 Conclusion and Boarder Impact
In this paper, we propose endow the current multimodal reasoning paradigm with the ability of learning from inside on the VideoQA task via Siamese Sampling and Reasoning (SiaSamRea), which contains two key parts: (1) siamese sampling to produce some sparse clips with similar semantics in the same video; (2) reasoning strategy to distill the interdependent knowledge between clips into the network. The reasoning strategy is composed of modules: (i) siamese knowledge generation to implicitly aggregate the inter-relationship of clips from the same video; (ii) siamese knowledge reasoning to infer soft label by using the predicted candidates of all clips and their inter-relationship Our proposed SiaSamRea finally can be jointly evolved by the soft label guidance and ground truth, which is evaluated on five VideoQA datasets demonstrating state-of -fe----ttthhtte art performance.
This work analyzes an interesting problem of how to learn from inside on the video question answering. Different from current methods mainly look for insights from outside data and powerful pretrained models. Our work proposes a totally different insight called learning from inside. In this paper, we observe the potential power of clips in the same video. Intuitively, the multiple clips should contain rich interdependent knowledge which is ignored by current ad advanced methods. To the best of our knowledge, we are the first one present learning from inside by using the knowledge between internal clips to assist in evolving the network training. Moreover, our method only teaches the network with interdependent knowledge during the training, which does not bring any extra burden (e.g., computation, memory and parameters) in the inference. Hence, it is light way to easily equip our method with other suitable networks on different multimodal tasks. We hope that our work will increase interest in the exciting field of learning from visually contextualized knowledge.
6 Acknowledgments
We thank all reviewers for providing the thoughtful and constructive suggestions. This work was supported in part by National Natural Science Foundation of China under Grant No.U1811461, in part by Natural Science Foundation of Guangdong Province, China under Grant No. 82030312002. and in part by the Major Program of Guangdong Basic and Applied Research No.2019B030302000
10

---

References
[1] A. Agrawal, J. Lu, S. Antol, M. Mitchell, C. L. Zitnick, D. Parikh, and D. Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31, 2015.
[2] E. Amrani, Ben-Ari, D. Rotman, and Bronstein. Noise estimation using density estimation for self-supervised multimodal learning. In AAAI, 2021.
[3] X. Chen, H. Fang. T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L., Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv, 2015.
[4] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. UNITER: Universal image-text text representation learning. In ECCV, 2020.
[5] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and -at---nndd--ggggee tasks via text generation. arXiv, 2021.
[6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
[7] C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In CVPR, 2019.
[8] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. ArXiv, abs/ 16066068847 2016.
[9] J. Gao, R. Ge, K. Chen, and R. Nevatia. Motion cationapppearacce co-memory networks for video question answering. In CVPR, 2018.
[10] L. Gao, P. Zeng. Song, Y.-F. Li, Liu, T. Mei, and H. Shen. Structured two-stream attention network for video question answering. In AAAI, 2019.
[11] D. Huang, P. Chen, R. Zeng, Q. Du, M. Tan, and C. Gan. Location-aware graph convolutional networks for video question answering. In AAAI, 2020.
[12] Z. Huang, Z. Zeng, B. Liu, D. Fu, and J. Fu. Pixel-BERT: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004. 00849, 2020.
[13] Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim. TGIF-QA: Toward apatio-tempores reasoning in visual question answering. In CVPR, 2017.
[14] J. Jiang, Z. Chen, H. Lin, X. Zhao, and Y. Gao. Divide and conquer: Question-guided spatio- temporal contextual attention for video question answering. In AAAI, 2020.
[15] Jiang and Han. Reasoning with heterogeneous graph alignment for video question answering. In AAAI, 2020.
[16] W. Jin, Z. Zhao, M. Gu, J. Yu, J. Xiao, and Y. Zhuang. Multi-interaction network with object relation for video question answering. Proceedings of the 27th ACM International Conference on Multimedia, 2019.
[17] J. Kim, M. Ma. T. Pham, K. Kim, and C. D. Yoo. Modality shifting attention network for multi-modal video question answering. In CVPR, 2020.
[18] T. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ArXiv, abs/1 009.609999990 2017.
[19] R. Krishna, K. Hata, F. Ren, I. Fei-Fei, and J. C. Niebles. Dense-captioning events in videos. 2017.
[20] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 2017.

---

[21] M. Le, V. Le, S. Venkatesh, and T. Tran. Hierarchical conditional relation networks for video question answering. CVPR, 2020.
[22] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for video-and-languaar learning via sparse sampling. In CVPR, 2021.
[23] Lei, L. Yu, M. Bansal, and L.. Berg. TVQA: Localized, compositional video question answering, In EMNLP, 2018.
[24] L. Li, Y.-. Chen, Y. Cheng, Z. Gan, L. Yu, and J. Liu. HERO: Hierarchical encoder for Video+language omni-representation pre-training. In EMNLP, 2020.
[25] X. Li, J. Song, L. Gao, X. Liu, W. Huang, X. He, and C. Gan. Beyond RNNs: Positional self-attention with co-attention for video question answering. AAAI, 2019.
[26] X. Li, X. Yin, C. L.
X. Hu, P. Zhang. L. Zhang. L. Wang, H. Hu, L. Dong, F. Wei, Y. Choi, and J. Gao. Oscar: Object-semantics aligned pre-training for vision-languee tasks. In ECCV, 2020.
[27] Y. Liu, M. Out, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and
V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv, 2019.
[28] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[29] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and Sivic. H0wto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[30] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. Vetito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-perffrmance deep learning library. In NeurIPS, 2019.
[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. ArXiv, abs/2 103.00020, 2021.
[32] M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NIPS, 2015.
[33] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele. A dataset for movie description. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3202-3212. 2015.
[34] P. H. Seo, A. Nagrani, and C. Schmid. Look before you speak: Visually contextualized utterances. In CVPR, 2021.
[35] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. VideoBERT: A joint model for video and language representation learning. In ICCV, 2019.
[36] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP, 2019,
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. 2017.
[38] D. Xu, 7. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, 2017.
[39] C. Yan, Y. Tu, X. Wang, Y. Zhang, X. Hao, Y. Zhang, and Q. Dai. Stat: Spatial-temporal attention mechanism for video captioning. IEEE Transactions on Multimedia, 22:229-241 2020.
[40] T. Yang, Z. Zha, H. Xie, M. Wang, and H. Zhang. Question-aware tube-switch network for video question answering. Proceedings of the 27th ACM International Conference on Multimedia, 2019.
12

---

[41] Z. Yang, Z. Dai, Y. Yang. J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. Xinet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
[42] Z. Yang, N. Garefa, C. Chu, M. Otani, Nakashima, and H. Takemura. Bert representations for video question answering. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1554, 2020.
[43] Z. Yang, N. García, C. Chu, M. Otani, Y. Nakashima, and H. Takemura. A comparative study of language transformers for video question answering. Neurocomputingg 445:121-133, 2021.
[44] Yu, J. Li, Yu, and Q. Huang. Multimodal transformer with multi-view visual representation for image captioning. IEEE Transactions on Circuits and Systems for Video Technology, 30:4467-4480, 2020.
[45] W. Yu, J. Liang, I.. Ji, I.. Li, Fang, N. Xiao, and N. Duan. Hybrid reasoning network for video-based commonsense captioning. ArXiv, abs/2108.02365, 2021.
[46] W. Yu, J. Zhou, W. Yu, X. Liang, and N. Xiao. Heterogeneous graph learning for visual commonsense reasoning. In NeurlPS, 2019.
[47] Y. Yu, H. Ko, J. Choi, and G. Kim. End-to-end concept word detection for video captioning. retrieval, and question answering. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3261-3269, 2017.
[48] Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao. ActivityNet-QA: A dataset for understanding complex web videos via question answering. AAAI, 2019.
[49] 7. Zha, Liu, T. Yang, and Zhang. Spatiotemporal--exuaal co-attention network for video question answering. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 15:1 18, 2019.
[50] Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L.-J. Wang, Y. Choi, and J. Gao. Vinvl: Making visual representations matter in vision-language models. ArXiv, abs/2101.00529, 2021.
[51] L. Zhu and Y. Yang. ActBERT: Learning global-local video-text representations. In CVPR, 2020.
[52] Y. Zhuang, D. Xu, X. Yan, W. Cheng, Z. Zhao, S. Pu, and J. Xiao. Multichannel atten- tion refinement for video question answering. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 2020.
13


Reviews:
[{"review_id": "a4aP7rkvVN0", "reviewer": "Reviewer_R2rx", "summary": "The paper addresses VideoQA and proposes a new approach: \"Siamese Sampling and Reasoning\". The method samples multiple sparse clips per video, an anchor clip, and several siamese clips with the same length but different starting frames. Then, pairwise similarities between these clips are computed and used to produce soft learning targets: Intuitively, all clips should produce similar predictions of the answer class. This self-supervised cross-entropy loss is combined with standard cross-entropy between predictions of the anchor clip and the ground truth labels. The method is evaluated on 5 VideoQA benchmarks and shows significant improvements.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\n- The method is novel in that it extends ClipBERTs sampling strategy of using several sparse clips with a new loss and new modules over ClipBERTs simple averaging strategy.\n\n- Related work has been analyzed and described thoroughly.\n\n- Strong results in combination with ablations suggest that the proposed method could be helpful for video understanding tasks.\n\n\n## Weaknesses\n\n### Major points:\n\n1- **Confusing description**: \nAuthors claim to learn \"inter-relationship among clips\" (line 12) and state that other methods \"consider each clip-text pair separately\" (line 36), \"each clip-text pair is independently encoded\" (line 107). This sounds to me like they would combine several clip-text data points in some way and integrate knowledge between data points. However, the method samples multiple combinations of frames for the same data point, therefore I find these main claims confusing. E.g. ClipBERT simply states in its abstract that \"a single or a few sparsely sampled short clips from a video are used at each training step\" which I find to be a much more fitting description.\n\n2- **Effect of pre-training**: Authors load pre-trained weights from ClipBERT and then pre-train again on TGIF-QA dataset. They claim that this has advantages (line 335), however, there is no ablation to support this claim: It would be helpful to see the results on MSRVTT-QA and MSVD-QA without pre-training on TGIF-QA dataset.\n\n3- **No comparisons to ClipBERT**: \n- ClipBERT results are missing from Table 5 (Performance on TGIF-QA), while they are added in Table 3 (Performance on MSRVTT-QA and MSVD-QA) and no reason is given on why these results are missing. This is important as ClipBERT reports better performance on TGIF-QA than the paper and has a very similar setup (same model, no more than 1 clip during testing, see ClipBERT paper table 8 second-last row.)\n\n- The ablations in Table 1 should be additionally done either on TGIF-QA or on MSRVTT-QA and compared to the ClipBERT result described above.\n\n4- **Hyper-parameters**: Authors use dataset-specific hyperparameters tuning (line 276) which could be another reason for some of the improvements over the ClipBERT method. What was the hyper-parameter tuning approach?\n\n5- **Table 1**: This table reports the ablation study on the effect of SKG and SKR components. However, there is no result with SKG alone. Also, the authors don't report the mean and std of multiple experiments. Would be great if the authors report mean/std and also results with only SKG. \n\n\n*Given the above concerns, it seems that the main improvement in this paper comes from taking ClipBERT and pretraining it on TGIF-QA before applying it to other datasets. This is contradicting to the story which reasons that the improvements are due to the proposed new modules.*\n\n\n\n### Minor points:\n\n1- Motivation is missing as to why the method is evaluated exclusively on VideoQA and not on other tasks like e.g. Video Classification. \n\n2- There are no details on pre-training and fine-tuning time complexity.\n\n3- Architecture details are missing: \"multimodal transformer is applied to produce the clip-text feature\" (line 133). I would assume the output is a single feature vector obtained from the CLS token, however, this should be specified directly. Otherwise, it would be very hard to reproduce the paper.\n\n**Typos**:\nThe paper is difficult to follow. The structure of the paper needs to be improved and also there are lots of typos and grammar errors:\n- 78 \"Not\" should be \"Note\"\n- -Figure 3 sentence 1 \"to be extracted\" sentence is missing a proper verb.\n- 129 \"all clips number\" is not a proper term, should be \"number of all clips\".\n- 129: \"Each clip ... is uniformly sampled ... frames\" should be rephrased to \"frames are sampled per clip\" or similar.\n- 136 \"dimension\" should be \"dimensions\"\n- 144 \"including\" is confusing here. Does this mean: siamese clips have semantics that are similar \"to\" the anchor clip?\n- 153: \"predictions on their predicted class probabilities\": Duplicate \"predict\" is incorrect\n- 225, 278: \"excavate\" means physically digging (e.g. with a shovel), would be better to replace it with \"use\" here. \n- 364 \"Boarder\" should be \"Broader\"\n- 375 \"look\" should be \"looking\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "WthQ6__bIwt", "reviewer": "Reviewer_QHNZ", "summary": "In this paper, the authors propose a network architecture to improve the Video QA task. The model learns and aggregates knowledge from siamese (nearby) frames and develop a reasoning strategy for integrating the inter-dependent knowledge between these contextual clips to make the final prediction. Their two modules, first, the siamese knowledge generation first learns inter-relationships among clips (anchor and siamese) and the second module, siamese knowledge reasoning that produces soft labels for self-supervised refinement during the training. They perform extensive experiments on five Video QA datasets to show the ability of their proposed SiaSamRea on both open ended QA and Multiple Choice QA and improve over the state-of-the-art models. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method, Siamese Sampling and Reasoning (SiaSamRea), has three main components. First, the sparse sampling strategy to get sparse frames and the siamese sampling strategy to get the siamese frames. Then, these frames are encoded using video encoders and a multimodal transformer generates multimodal features which are used to produce contextual relationships between the anchor frames and the siamese frames from the same video. Finally, the knowledge reasoning module generates soft labels based on the similarities between frames acquired in the generation module. The model is trained using cross entropy loss or the multi-margin ranking loss based on the task at hand. The proposed method is novel and the paper is well written. \n\n1. Are the weights for the Video Encode /Text Encoder and Multimodal transformer all initialized using the ClipBert model and then fine-tuned on the respective datasets ? It is good to mention what these encoders are eg. resnet 50, 12 layer transformer etc. \n\n2. In section 3.3.1, the matrix A in line 161 is NxN. Is N the total number of anchor and siamese samples in the full video ? f_anchor_1, f_siam_11, f_siam_12, f_siam_13, f_anchor_2, f_siam_21, f_siam_22, f_siam_23 ……. And so on? \n\n3. Did the authors try using the anchor as the center frame and having siamese frames near to both sides of the center frame? \nCurrently, from the figure and description it seems the siamese frames are samples only at (t_anchor + t_s) locations. \n\n4. During training, as there are more clips/frames for each video sample, how much does it increase the training time and memory usage compared to, for eg.  ClipBERT? \n\n5. Will your method underperform/overperform if attention over frames is done in dense sampling or following methods similar to Long term feature banks [1]?\n\n[1] Wu, Chao-Yuan, et al. \"Long-term feature banks for detailed video understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n\n##Post Rebuttal\n\nI have read the authors' response and they address my concerns. I maintain my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not discussed any limitations or the social impact of their method or the domain of VideoQA in general. There are some things that could be important to consider while designing models for these datasets and problems. Are the datasets biased towards some specific answers given the questions (for eg, if there’s perching in the question, would the answer always be bird?). This is an important thing to consider if the training and test distributions are different and to reduce bias in learning. Is the model looking at the image to answer these questions or just overfitting to the textual bias? Also, due to pre-training on large scale datasets which are also potentially biased (racial, gender etc.), this might get propagated in the downstream task training and  hence not suitable if such models are to be deployed in real settings. The authors should consider these limitations and discuss them in the paper. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "80FKph-Vi26", "reviewer": "Reviewer_o6HV", "summary": "This paper proposes a Siamese Sampling and Reasoning network (SiaSamRea) to model the interdependent semantic across clips in a video. The contribution of the paper is to employ Siamese sampling and calculate the similarities between the multimodal features of clip-text pairs. The Siamese knowledge is then used to generate soft labels which guide the learning process. Experimental results on five Video QA datasets showed the effectiveness of the proposed method against existing works.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) I like the idea of modeling the interdependent relationship of internal clips. The proposed method is well-motivated and straightforward for implementation.\n\n(2) The proposed method shows improvements across multiple Video QA datasets\n\nWeaknesses:\n\n(1) Although the idea is interesting, the current implementation appears to be an extended version of ClipBERT with Siamese network.\n\n(2) There are several missing details about the model that the authors should explain more clearly:\n\n•\tHow many anchor clips are used in the proposed model? How does the number of anchor clips affect the performance?\n•\tIn Eq. 5, what is the soft learning target Q used for? \n\n(3) Experiments:\n- Why didn’t the authors report the performance of ClipBERT on TGIF-QA dataset in Table 5. As per results from ClipBERT paper, it outperforms the proposed method on most tasks, as result for the Count task is not available. It is important to compare the proposed method with ClipBERT on the TGIF-QA as it is the most prominent dataset for Video QA. \n\n- As SiaSamRea shares some common components with ClipBERT, please explain the performance degrade on the TGIF-QA against ClipBERT.  \n\n- The examples in Figure 4 do not show the interdependent relationship between internal clips in the video. Please provide more diverse showcases.\n\n- Ablation studies are not sufficient. Please justify your choices of hyperparameters such as the number anchor clips, the number frames in each clip, the value of the balance term, and the integrating weight.\n\n - One of the downsides of the Siamese networks is that it is quadratic in time. Please provide more analysis on the memory usage and computation cost of the proposed model.\n\nMinor comment: Page 6, Line 190: multi-label classification -> multi-class classification\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "-bNsEy2PreK", "reviewer": "Reviewer_a2XV", "summary": "This paper proposes a novel approach, Siamese Sampling and Reasoning (SiaSamRea), to give self-taught guidance information as soft-labels to train the models via sampling several similar short-clips within the same video for the task of video QA. The authors provide experimental results to validate the effectiveness with new state-of-the-art performances on five benchmark datasets: MSRVTT-QA, MSVD-QA, How2QA, ActivityNet-QA and TGIF-QA.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The task of video QA is one of important and interesting problems in visual-linguistic AI. Since computational cost can be large for video understanding to handle arbitrary questions related to the video contents, it is inevitable to take certain sampling strategies. For example, one of recent works, ClipBERT, provides good approaches with sparse clip sampling. On the other hand, the proposed approach utilizes more sample clips that are similar but not same. With the aggregated features of the samples, the authors propose the methods to improve the models without supervision. The inductive bias of this work seems that “clips with high visual similarities are expected to have more consistent predictions on their class probabilities.” It can be achieved via Siamese sampling, sampling more short-clips near the anchor clip. From the assumption, the experimental results seem to show new state-of-the-art performances on some benchmark datatsets such as MSRVTT-QA, MSVD-QA, How2QA and ActivityNet-QA.\n\nHowever, this paper has some weak points. \nFirstly, the proposed methods seem to be based on ClipBERT [22] mostly. Following the manuscript, most parts of the architectures come from ClipBERT including video encoder, text encoder, and multi-modal transformer as a model of initial weights. I think the result on ClipBERT should be reported on all of the tested benchmark datasets. But, the results on MSRVTT-QA only are reported in Table 3. In this case, it is not clearly validated that the proposed method gives positive impacts. Moreover, the result on ClipBERT on TGIF-QA was reported in [22], and they are better than the result of the proposed approach.\nSecondly, it is not clear that all of experiments performed fairly on all of comparative methods. The models in this paper are pretrained with TGIF-QA dataset from ClipBERTs as initial models, and the fine-tuning is done for the downstream tasks. On the other hands, it seems that the other methods were trained with their datasets and the strategies. If it is not easy to be performed properly, it would be better to clarify their experimental settings briefly in the manuscript.\nThirdly, it would be more persuasive to survey self-supervised learning methods in related work section.\n\nPersonally, I think the term Siamese is not good choice on this concept. It might lead to confusion.\nAs a suggestion, it would be better to clearly formulate and explain the proposed methods if finding and utilizing analogous machine learning frameworks, e.g., multi-view learning [1*,2*]. Probably, it will help the authors figure out pros and cons, the potentiality and the limitation, and find better directions to improve.\n\n\n\n* Minor\n   -\tline 139, spare -> sparse\n\n\n[1*] H. Su et al., Multi-view Convolutional Neural Networks for 3D Shape Recognition, ICCV 2015.\n\n[2*] B. Xiong et al., Multiview Pseudo-Labeling for Semi-supervised Learning from Video, Technical Report, https://arxiv.org/abs/2104.00682.\n\n----------------------------\n### Post Rebuttal\n\nI have read all of other reviews and the authors' response. \nThe rebuttal resolves most of my concerns. \nHowever, I think that there is still remained necessity to improve the manuscript in presentation, discussion and reproducibility.\nI keep my score as it was.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no limitations and societal impact in the manuscript.\nHow about mentioning the potential risks of the proposed video QA systems being trained with social biases on image data from the public or the industry?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "Basically, Visual Question Answering systems including video QA provide answers based on arbitrary questions, especially open-domain QA.\nIf the questions include certain ethical values, how does this system generate answers?\n"}}]

Author rebuttal:

Meta-review: After the authors’ rebuttal, intensive discussion followed between reviewers, and finally the reviewers’ consensus has reached to the reject opinion for this submission at this time. All agree that the main idea of this work is interesting, and they do not want to be strongly against this work. However, the reviewers also acknowledged that there is much room for improvement to be published. For example, the paper needs more clarifications and details about where the improvements come from, ClipBERT should be compared in different scenarios, and better benchmark datasets may be necessary. 
Decision: Accept (Poster)