Abstract

1
Introduction
Video question answering (VideoQA), which is designed to predict an answer to a given question based on a relevant video clip, requires comprehending both visual and linguistic content. This has made VideoQA an ideal testbed to evaluate current machine learning mod- els, and researchers have made great efforts to advance the field. For example, TVQA [[ 3 ] presents a large-scale dataset and a model that leverages Faster-RCNN [ [ 2 ]] and LSTMs [ [ 2 ] to process visual and language inputs, and the use of attention mechanisms [ [ 20 ] has also achieved a great success [ 60 , 62 , 651 ]. Recently, a new research direction in VideoQA has emerged, i.e ., external knowledge-based VideoQA [Ω, ✿], which requires information that 2 cannot be directly obtained from the videos or the question-answer (QA) pairs, and thus, can- not be learned from the dataset. In this task, therefore, a model needs to refer to knowledge from external sources. This configuration may be closer to real-world question answering. Knowledge for VideoQA, including external knowledge-based one, can be obtained in different ways: some knowledge can be learned from the videos and QA pairs in the dataset, and other knowledge can be extracted from a (external) knowledge base. Existing approaches use a dedicated set of sentences, each of which is associated with its respective QA pair [ 8 ] or a block of text found in the Internet [ A , [ ], as knowledge base in a specific domain, which are retrieved by the model during inference. This means that the knowledge base can be potentially replaced without re-training. In this case, a natural question arises here: How much is a trained VideoQA model's knowledge generalised for different domains? In this paper, we study the transferability of knowledge learned in VideoQA. Specifi- cally, we split the knowledge into two categories (Figure 1): domain-specific and domain- agnostic . The domain-specific knowledge is related to a particular domain of videos in a VideoQA dataset. This type of knowledge is acquired for each domain via training. On the other hand, the domain-agnostic knowledge involves (i) the so-called tacit knowledge, such as how to understand videos and questions as well as how to retrieve the desired piece of knowledge from the knowledge base and (ii) the so-called explicit knowledge that is still valid for different domains, such as common-sense. Based on this distinction, we propose a knowledge-oriented transfer learning for better generalisation of VideoQA models. We study the influence of knowledge across datasets and mitigate the inter-dataset information gap by effectively transferring the domain-agnostic knowledge. Our main contributions are summarised as follows: 1) We show that, like humans, domain-specific and domain-agnostic knowledge also ex- ists in VideoQA. Also, the domain-agnostic knowledge is transferable and can bring considerable performance improvement. 2) For transfer learning experiments on VideoQA, we construct a new dataset, KnowIT- X, containing 21,412 QA pairs, each of them annotated with knowledge. 1 The video clips are sampled from a TV show, Friends , which is a different domain from the standard model [ 8 ]'s with The Big Bang Theory . 2
Related Work
Video question answering Traditionally, temporal visual information and language are processed with deep neural network (DNN)-based models and fused to predict the correct answer. For example, LSTMs [ L ] are used to embed the temporal visual and textual features in [[5], and attention mechanisms [04], which allows to focus only on specific parts of the input, bring significant improvements in [ å , å ], å ]. Other work [ å ], å ]] applies Transformers [ 8 ] to capture the information from videos. For better fusion of independent visual and textual sources, Hirota et al. [[[0, [[] ] propose to use the textual representations to understand the visual sources. Besides the independent processing of vision and language, other approaches [ 0 , [ 2 ] model the relationships between objects. Recent works like  [ 6 ]] also focus on the VideoQA without much human annotation. Another direction is to retrieve external information from knowledge bases (KB), which successfully extends the linguistic features and steers the model to a specific part of the visual Figure 1: Examples of transfer learning on different domains of VideoQA. The two datasets (source and target) have different domain-specific knowledge but share the same domain- agnostic knowledge. We argue that the latter can be transferred across datasets. content. For example, in visual question answering (VQA), a structured KB like ConceptNet [03] or Freebase [0] is adopted as extra inputs in [03, 06, 07] and a generic unstructured KB is investigated in [[8]. For VideaQA, KnowIT VQA [8] is the first unstructured video-based dataset built by humans.  ROLL [闻] leverages online knowledge to answer questions about video stories, showing the great potential of knowledge-based models in VideoQA. Transfer learning in vision and language Transfer learning has been rarely investigated in VideoQA, but it has been addressed in the related VQA tasks. Hu et al. [ [ 3 ] study transfer learning between open-ended VQA datasets by a probabilistic model. Xu et al. [ 29 ] learn and make use of joint features across different modalities. Chao et al. [ 6 ] propose a domain adaptation method to deal with the cross-dataset mismatch on images, questions, or answers. However, these approaches are not directly applicable to VideoQA due to the much richer information contained in videos.  In addition, more emphasis has been put on processing the dataset itself, rather than studying the knowledge learnt by the model, which gives a more general solution to diverse domains of datasets. Therefore, this paper proposes a novel knowledge-oriented transfer learning method for VideoQA. To the extent of our knowledge, this is the first work that studies the transfer of knowledge in VideoQA.
Methodology
3
Figure 2 outlines the training pipeline of our proposed transfer learning method. First, in the pre-training stage, we use a source dataset to train a knowledge retrieval module of a stan- dard knowledge-based VideoQA model (Section 3.1 ). In the knowledge retrieval module, a question and its candidate answers are used to retrieve the most relevant piece of knowledge from a KB. We design the pre-training stage so that the knowledge retrieval module learns to mitigate the domain-specific knowledge and focus on the domain-agnostic knowledge by pre-processing the source dataset with a domain-specific entity tagger (DET) (Section 3.2). Then, in the transfer learning stage, we pre-process the target dataset with DET and incorpo- rate a data augmentation module (DA) (Section 3.3). Using the domain-agnostic knowledge acquired in the pre-training stage as a medium, we finetune the pre-trained knowledge re- trieval module with the pre-processed target samples (Section 3.4 ). Finally, the output of the knowledge retrieval module, together with the visual features, question, and candidate answers, is used in a video reasoning module to predict the correct answer (Section 3.5).
3.1
Knowledge-based VideoQA model
We use ROCK [ 0 ] as the backbone VideoQA model. ROCK differs from standard VideoQA models [ 0 , [ 4 , [ 5 ] in that it explicitly uses external knowledge obtained from a KB, making it easier to study the problem of transferring knowledge across different domains.  ROCK takes a video clip, a question, and a set of candidate answers as input, and 1) retrieves relevant knowledge to answer the question from a KB and 2) predicts the correct answer by leveraging the input and the retrieved knowledge.  Specifically, ROCK presents a three-stage pipeline, including (a) KB construction, (b) knowledge retrieval, and (c) video reasoning. The KB is constructed with annotated knowledge from the original dataset [ 0 ]. The knowledge retrieval module is a BERT network [ ◘ ] trained to match the most relevant knowledge in the KB given a question and candidate answers. Finally, the video reasoning module fuses the retrieved knowledge with the rest of the language and visual inputs to predict the correct answer. For further details, we refer curious readers to [ 8 ].
3.2
Domain-specific entity tagger (DET)
VideoQA models usually have strong in-domain generalisation across two sets of QA pairs with identical distributions [ [ 3 ]. However, in a transfer learning setup, a model is pre-trained on a source dataset and evaluated on a target dataset, where these datasets are from different origins and present different distributions. This leads to a large gap between the knowl- edge contained in the two datasets in terms of scenes, plots, and vocabulary.  Thus, the domain-specific knowledge is specialised for a certain dataset (or domain) and may hinder the transfer to out-of-domain data [ [ 1 ]. To mitigate the inter-domain difference, especially in the language modality, and facilitate more effective learning of domain-agnostic knowledge, we propose a domain-specific entity tagger (DET) to recognise and tag the domain-specific knowledge with the named entity [ 20 ] they belong to. As the entity names are more gen- eral expressions, we relieve the influence brought by the domain-specific knowledge in the source dataset so that the model does not need to know them.  In this way, the transfer of domain-agnostic knowledge will be much easier by recognising the domain-specific knowl- edge. Specifically, for a question q in the dataset, such as ' Why was Chandler acting weird? ', we recognise and tag the entity Chandler as Person . To maintain the original semantics and the grammatical structure, the tagged entity ( e.g., Chandler ) and its entity type ( e.g., Person) are inserted back into the original sentence in an appositive form. The question DET(q) after this entity tagging becomes: 'Why was Chandler, a person, acting weird?'. We apply DET to question q , candidate answers { a i } i , and knowledge instance k in both the source and the target dataset. We denote the set after applying this by DET( X ), i.e., DET( X ) = {(DET(q), {DET( a i )} i , DET(k) | (q, { a i } i , k ) ∈ X } ,
where X denotes any VideoQA set.
3.3
Target data augmentation (DA)
When applying transfer learning from a source to a target dataset, the latter may not contain enough samples to learn sufficient domain-specific knowledge in the target domain, which may cause overfitting. To address this problem, we augment the training set T in the target dataset before transferring the pre-trained model.  To augment the training set we propose to apply back translation [ [ [ 0 , [ 2 ], which translates a natural language sentence into a pivot language with the function Translate( · ), and then translates it back to the original language with the function BackTranslate( · ). With this technique, we incorporate twice as many sam- ples as the original dataset, with a different structure but maintaining the original semantics. Also, for extremely small-scale datasets, back translation can be applied several times using multiple pivot languages. Formally, for any text s ( i.e. , a question, an candidate answer, or a knowledge instance) in the training set, spv = Translate(s) (2) s' = BackTranslate(spv), where spv is the sample in the pivot language and s ′ the augmented sample. As some s ′ , can be almost the same as its original ( s ′ ι ≈ s i ), we identify them with a pre-trained BERT network [ 0 ] and similarity threshold α to remove them. We incorporate the new s ′ to the target training set T , creating DA( T ) = T ∪ T BK , where T BK is the set of back-translated triplets of a question, a set of candidate answers, and a knowledge instance, after removal. We apply DA to all the questions and candidate answers in the target training set.
3.4
Domain-agnostic knowledge transfer for knowledge retrieval
With the samples tagged with domain-specific knowledge and augmented by back transla- tion, we next introduce the transfer of the domain-agnostic knowledge. Our VideoQA model is pre-trained on the training set S of the source dataset and then transferred to the target dataset.  During pre-training, the knowledge retrieval module is trained to locate the most relevant knowledge instances kg in the source KB given the input source question q and source candidate answers { a } . Using ρθ(kg | q,{ a } ), which denotes the probability of kg being relevant to q and { a } parameterized by θ , this can be formulated as a maximisation problem of the expected log-likelihood: θρre = arg max E[log(po(kg | q, {a}))],
where the expectation is computed over all (q, { a },kg) ∈ DET( S ). In the transfer learning stage, the pre-trained knowledge retrieval module is finetuned to find parameter θ FT with the same training strategy as Eq. (3) but the expectation is computed over all (q,{a},kg) ∈ DA(DET( T )) and the parameters are initialised by θ ρ re .
Answer prediction on the target domain
3.5
To predict the final answer, we extract visual features v from the input video clip in the target dataset and fuse them in the video reasoning module together with question q , candidate answers { ai } i , and retrieved knowledge kg = arg max kg ρθετ (kg | q, { a} ) by the transferred knowledge retrieval module. We extract v at three different scales of information: image features from video frames using ResNet50 [ [ ], facial features with recogniser [ [ 3 ], and captions generated with [ Ω] . The textual data ( i.e., q , { ai } i , kg ) is concatenated and encoded in single vector u i per candidate answer for i = 1,...,N a ( N a is the number of candidate answers), as in [ 8 ]. The visual and language features are concatenated and projected with a fully-connected layer to obtain an answer score.  The predicted answer is the one with the highest score. The video reasoning module is trained with a multi-class cross-entropy loss.
KnowIT-X VQA Dataset
4
We use KnowIT VQA [ 8 ] as source dataset, as it contains annotated knowledge for each sam- ple in VideoQA. As there are no other comparable datasets with knowledge for VideoQA to use as target, we collect a new one following KnowIT VQA framework. We name it KnowIT- X VQA (X for transfer learning). While KnowIT is collected from the TV show The Big Bang Theory , KnowIT-X is from another popular TV show: Friends. We choose Friends be- cause it shares strong similarities with the original KnowIT VQA dataset in terms of format, video length, and audience, whereas it is different in terms of plots and topics, making it ideal for transfer learning. Additionally, to study transfer learning at multiple scales (trans- fer learning is commonly applied to small-scale datasets), we create three different versions: KnowIT-X full, KnowIT-X v-5k, and KnowIT-X v-3k, with all the collected samples, 5,092 samples, and 2,900 samples, respectively. Details are shown in Table 1. Some examples of KnowIT-X are shown in the supplementary material. Video clip extraction    We obtain videos and subtitles from the original DVDs. We use 202 episodes, distributed in all 10 seasons of Friends. We divide each episode into 20-second clips by scene, where each scene is aligned with episode transcripts available online. 2  We acquire 12, 176 clips in total. Annotations   Following KnowIT, for each video clip, we ask workers on Amazon Me- chanical Turk 3 to write a question, four candidate answers (one correct), and the associated
Table 1: Comparison between original KnowIT and KnowIT-X datasets. The three versions (v) of KnowIT-X are designed to study transfer learning on knowledge-based VideoQA. Figure 3: Vocabulary clouds for KnowIT and KnowIT-X VQA datasets. knowledge that makes the answer correct (see examples in Figure 1 ). After two sweeps of all video clips, we obtain 21,412 samples, which we split into 17,583 for training, 1,748 for validation, and 2,081 for test sets. For the smaller versions v-5k and v-3k, we randomly re- duce the training set to 4, 269 samples and 2, 432 samples, respectively. The same reduction is also applied to the test set and the validation set. The annotation process took us 2 months. Domain comparison While KnowIT is from the show The Big Bang Theory with scientific topics, KnowIT-X is focused more on the relationships among characters. This difference can significantly influence the audience's impressions and interests, causing a distribution gap between the annotations.  To show this gap, we first compare the vocabularies.  As illustrated in Figure 3, the vocabularies differ a lot. Additionally, we present the distributions of question types in Figure 4 , where the type is defined as the first word of each question. With more 'who' questions, KnowIT-X is more focused on the vision than KnowIT, which is predominately focused on knowledge, with a larger amount of 'why' questions. More comparisons between KnowIT and KnowIT-X are presented in the supplementary material.
Experiments
5
We use PyTorch [ [ 9 ] to implement our models. The hyperparameters are set following [ 8 ]. In the DET module, the specific entity names include 18 different entity types. 4  The DA module employs German as the pivot language and α = 0.998. Evaluation metrics Following [ 8 ], we report results both on the knowledge retrieval and the video reasoning modules. Knowledge retrieval results are reported as recall at k (R@k) with k = 1,5,10, and median rank (MR). Video reasoning results are reported in terms of
Table 3: Knowledge retrieval module results on KnowIT-X v-3k and v-5k. Each training configuration is computed with and without DET and DA. accuracy.  For the knowledge retrieval, we use MR as our primary metric and R@5 as the secondary metric, as the top-5 retrieved sentences are the ones used in the video reasoning module.  Additionally, note that as MR and recall do not consider the potential similarities between other highly related samples in the knowledge base, the evaluation of the knowledge retrieval may not be representative of the real improvement on the final VideoQA task. Domain-agnostic knowledge transfer with DET First, we evaluate the domain-agnostic transfer learning on the knowledge retrieval module. Table 2 shows results on KnowIT-X v-Full. We compare different learning methods: direct learning on a single dataset (KnowIT, KnowIT-X, or both), and transfer learning with and without our proposed DET. Direct train- ing on the original KnowIT (row 1) shows the worst performance by a large margin, con- firming the big domain gap between the two datasets. Transfer learning without DET (row 4), which is equivalent to standard finetuning, does not improve results over direct training (row 2), verifying that in this case the model is not able to discern between domain-specific and domain-agnostic knowledge. However, when we incorporate our proposed DET (row 5), results are successfully improved, obtaining the best R＠5 and MR. Direct training on both datasets (row 3) performs similar to our method, but it requires more time and memory. 5 Small-scale knowledge transfer The benefits of using the proposed transfer learning are even more obvious on the smaller versions of the dataset. Results on KnowIT-X v-3k and v-5k are reported in Table 3.  We apply DET and DA to all the training configurations, obtaining consistent improvements and indicating that the proposed method captures the shared knowledge between different datasets. The only exception is when we train directly on both datasets (KnowIT and KnowIt-X), as the model can obtain enough information at training time at the expense of a higher computation cost.  Even so, our proposed transfer learning method obtains the best performance both in KnowIT-X v-3k and KnowIT-X v-5k.
Table 4:  Knowledge retrieval results using different DET methods. edge). Image features are used as the vision source in this experiment.
Results with other metrics are presented in the supplementary material. We compare different ways of inserting the recognised named entities with DET analysis DET and show that the appositive tagging is better than other tagging methods. In particular, we evaluate Mask-out , in which the detected entity is removed and replaced by its label (e.g. Why was person acting weird?), and Hyphen separator , in which the entity and its label are separated by the character "-" (e.g. Why was Chandler-person, acting weird? ). Results are reported Table 4 for the model pre-trained on KnowIT and transferred to KnowIT-X v-Full. Video reasoning direct results   We evaluate the standard ROCK model [ 8 ] ( i.e. , without transfer learning) on the three versions of KnowIT-X and compare the results with the orig- inal KnowIT dataset. We use image features for the vision; QA, and subtitles (Subs) for the language; and retrieved and ground truth knowledge for the KB. All the models are trained from scratch.  Accuracy is shown in Table 5.  We observe that 1) the accuracy when only using questions and answers (row 1) is lower on KnowIT-X, indicating that the language bias is not as strong in KnowIT-X, 2) the small-scale datasets perform significantly worse than its full counterpart, indicating the importance of training data. 3) the performances with retrieved knowledge (row 4) surpass the one without knowledge (row 3) by a large margin, which demonstrates that the retrieved knowledge predicts the correct answers. Video reasoning transfer results   In Table 6, we show the video reasoning results on the KnowIT-X v-3k, v-5k and v-full with different scales of visual information. We compare the full transfer learning method against variations with and without DET and DA, and the direct learning method. It can be seen that each strategy contributes to improving the final VideoQA accuracy.  When both DET and DA are included, the result is the best.  Also, among all the vision sources, captions usually help acquire the best performance. Additionally, in Figure 5 , we show a qualitative comparison between direct and transfer learning. In direct learning, the retrieved knowledge is too much focused on the domain-specific entity 'hat' and has limited
ow does Phoebe know the man wearing the hat? : They used to attend art class together. A2: They used to mug people on the streets. : They used to play in the same band. ey used to go to the same high school. wed Knowledge: Phoebe, a person. tells A2: They used to mug people on the streets. Figure 5: Example results: (left) direct learning, (right) proposed transfer learning. contributions to the final answer. On the contrary, transfer learning successfully recognises it as a domain-specific entity, putting more emphasis on the relationship between people. The retrieved knowledge is highly connected with the question and the correct answer.
Conclusion
6
We proposed a knowledge-based transfer learning framework for VideoQA. We divided the knowledge learned by the VideoQA models into two categories and achieved better transfer results by mitigating the influence of the domain-specific knowledge and transferring the domain-agnostic knowledge. Back translation was applied to augment small-scale datasets. By constructing a new knowledge-based VideoQA dataset, the experimental results showed that domain-agnostic knowledge can be transferred adaptively from the source dataset and boost performance in the proposed framework. Acknowledgements This work was supported by JSPS KAKENHI No. JP18H03264 and JP20K19822, and JST ACT-I.