VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
Authors: Zineng Tang
Jaemin Cho
Hao Tan
Mohit Bansal
Venue: NeurIPS 2021 Poster (2021)
Keywords: video-language knowledge distillation
visual grounding
video-text transfer

Abstract: Since visual perception can give rich information beyond text descriptions for world understanding, there has been increasing interest in leveraging visual grounding for language learning. Recently, vokenization (Tan and Bansal, 2020) has attracted attention by using the predictions of a text-to-image retrieval model as labels for language model supervision. Despite its success, the method suffers from approximation error of using finite image labels and the lack of vocabulary diversity of a small image-text dataset. To overcome these limitations, we present VidLanKD, a video-language knowledge distillation method for improving language understanding. We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset. To avoid approximation error, we propose to use different knowledge distillation objectives. In addition, the use of a large-scale video-text dataset helps learn diverse and richer vocabularies. In our experiments, VidLanKD achieves consistent improvements over text-only language models and vokenization models, on several downstream language understanding tasks including GLUE, SQuAD, and SWAG. We also demonstrate the improved world knowledge, physical reasoning, and temporal reasoning capabilities of our model by evaluating on the GLUE-diagnostics, PIQA, and TRACIE datasets. Lastly, we present comprehensive ablation studies as well as visualizations of the learned text-to-video grounding results of our teacher and student language models.

VIDLANKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
Zineng Tang
Jaemin Cho Hao Tan Mohit Bansal
UNC Chapel Hill
(terran, jmincho, haotan, mbansal)@cs. enc.edu
Abstract
Since visual perception can give rich information beyond text descriptions for world understanding, there has been increasing interest in leveraging visual grounding for language learning. Recently, vokenization [68] has attracted attention by using the predictions of text-to-image retrieval model as labels for language model supervision. Despite its success, the method suffers from approximation error using finite image labels and the lack of vocabulary diversity of a small image-text dataset. To overcome these limitations, we present VIDLANKD, a video-language knowledge distillation method for improving language understanding. We train multi-modal teacher model on a video-text dataset, and then transfer its knowledge student language model with text dataset. To avoid approximation error, we propose to use different knowledge distillation objectives. In addition, the use of large-scale video-text dataset helps learn diverse and richer vocabularies. In our experiments, VIDLANKD achieves consistent improvements over text-only language models and vokenization models, on several downstream language un- derstanding tasks including GLUE, SQuAD, and SWAG. We also demonstrate the improved world knowledge, physical reasoning, and temporal reasoning capabil- ities of our model by evaluating the GIUUF-diagnosties, PIQA, and TRACIE datasets. Lastly, we present comprehensive ablation studies as well as visualiza- tions of the learned text-to-video grounding results of our teacher and student language models.'
Introduction
Language learning can be aided by grounded visual cues, as they provide powerful signals for modeling vastness of experiences in the world that cannot be documented by text alone [5; 29; 4]. While the recent trend of large-scale language model pretraining indirectly provides some world knowledge from text, most large text corpora (c.g.. Wikipedia) do not provide enough multi-modal grounding information. Previous works have explored multiple ways of grounding language to visual information such constructing common vector space [38; 7] and supervising the model with token-wer wise generated vision labels [68]. However, the widely-used image-text datasets (e.g., MS COCO [48]) are much smaller than text-only corpora in terms of word counts and vocabulary diversity for language learning.
The recent method of 'vokenization' [68] is promising initial step towards addressing this problem by supervising language models with weakly-aligned vision-language groundings. Firstly, an image- text matching model retrieves corresponding image each text token in sentence. Then language model learns to predict the selected image (called 'voken') )) for each text token. This can be seen as knowledge distillation (KD) process [33] from vision-language grounding model to a language
Code and models: tttps://githin wttpttttuu..mmmmmeeeeeeee wwtmm//iinnntaan///e VidLLnnD
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

(a) Cross-modal Pretraining
(b) Knowledge Distillation
Multi-modal Dataset (D,r)
Text Dataset (D.)
eggs putting pat Use your spatill get all the way acound.... ...
A large body of Westem Chusakea liceture in the Karmada 1200) produced during empire's reign (913 period, sssally consilered Kanuala,
Indio, Kantada Inerature from
Video
Text
Distillation
Freeze
Parameters
Teacher LM
Teacher LM
Student LM
Figure I: Overview of the proposed VIDLANKD method. We first pretrain a teacher language model on multi-modde dataset (Sec. 3.2). Then we distill the knowledge of the teacher model (weights frozen) to student language model on text dataset (Sec. 3.3).
model. Although the voken classification task helps the language model to improve on natural
language understanding (NLU) tasks, there exist several limitations: (1) images cannot faithfully convey word meanings that require more activity-based and physical commonsense knowledge. (2)
the voken supervision suffers from approximattion/quaiizttioo error of the text-to-image retrieval.
To address these problems, we propose novel Video-and- and-Language Knowledge Distillation method, named VIDLANKD. Our teacher model consists of a video encoder and a language encoder. They are jointly trained with video-language contrastive learning objective and masked language modeling (MLM) objective on multi-modal dataset (see Fig. 1). Then, we transfer the knowledge of the frozen teacher language encoder a student language model by minimizing the distance between contextualized text representations of two models on text dataset. For this, we propose to use different KD objectives including neuron selectivity transfer (NST) [34] and contrastive representation distillation (CRD) [71] that avoid the approximation error from voken assignments [68]. For tross-modes pretraining our teacher model, we use How 100M [54], large-scale video dataset which has more diverse vocabulary and richer world commonsense (e.g., physical and temporal) knowledge compared to MS COCO image dataset.
In our experiments, student language models learned with the proposed video-language KD objectives outperform the baseline text- rext-pretr language models and the models distilled with vokeniza- tion, on several diverse natural language understanding benchmarks including GLUE [73], SQuAD [61], and SWAG [79]. We also show comprehensive ablation studies on video encoders. student KD objectives, teacher pretraining objectives, and video vs. image-based pretraining. Further- more, we empirically illustrate that our model successfully learns linguistic world knowledge and physical/temporal commonsense abilities from video, by showing improved performances the GLUE-diagnostics [73], PIQA [6], and TRACIE [82] datasets.
Overall, our contributions are: (I) a novel cross-modal knowledge distillation method for improving natural language understanding, (2) using rich video-text data which can overcome the limitations of image vokenization, (3) empirical improvements on several language understanding benchmarks and studying different knowledge distillation methods, and (4) analysis on lingussie/pyysicallemmmo knowledge learned from videos and ablation studies on the effectiveness of proposed components.
Related Work
2.1 Knowledge Distillation
Knowledge distillation (KD) [33] is the process of transferring knowledge from a teacher model to student model. has been successfully used in a wide range of applications, such as machine translation [40], visual recognition [31], speech recognition [10], and recommendation systems [69]. Recent works advanced the field of knowledge distillation by proposing new architectures [77; 80; 1; 55] and objectives [34; 14].
While many KD works study the problem of knowledge transfer within the same modality, cross- modal knowledge distillation [27; 20; 71] tackles the knowledge transfer across different modalities. Gupta et al. [27] transfers the knowledge of a model trained with RGB images to another model for depth maps and optical flow. Do et al. [20] proposes KD method for visual question answering

---

[2], where the trilinear question-answer) relational representation of teacher model is transferred to bilinear (maae-question) student model. Tian et al. [71] combines contrastive learning and knowledge distillation to improve the knowledge transfer between different modalities. Our VIDLANKD transfers the knowledge of a multi-modal teacher model learned from video dataset to student language model that tackles natural language understanding tasks.
2.2 Language Pretraining
Large-scale pretraining of contextualized language models has seen huge success in natural language processing in recent years. ELMo [57] proposes to pretrain and fine-tune a large recurrent language model, which improves performance
diverse of downstream natural language processing tasks. BERT [19] improves the scalability of the pretrain/fine-tune framework by using a transformer [72] language model with masked language modeling objective. Since then, pretraining of transformer language models has been extensively explored [49; 78; 44; 22; 64; 59; 16] for various natural language understanding [61; 74; 79; 73] and generation tasks [24; 62; 61; 63].
2.3 Multi-modal Pretraining
Following the success of language pretraining with transformer models, pretraining of image-text [67; 51; 13; 47; 83; 45] and video-text [66; 54; 85; 53; 46; 70] multi-modal transformers have achieved improvements on numerous multi-modal downstream tasks [2: 76; 84). The multi-modal transformers take both visual and textual inputs and are pretrained on image-text or video-text pairs with multi- modal masked language modeling objectives. Despite the success on multi-modal downstream tasks, Tan and Bansal [68] finds that the multi-modal pretraining does not improve (and sometimes even harms) the language understanding performance. This is because the scale and diversity of text vocabulary of image-text and video-text datasets usually smaller than those of text datasets. To utilize the rich vocabulary of text dataset, our VIDLANKD transfers the knowledge of pretrained multi-modal model to student language model with large text dataset.
2.4 Visually-Grounde Language Learning
series of works explore using visual information aid language understanding and generation tasks including co-reference resolution [42; 15], machine translation [81], bilingual lexicon learning
[39], and multi-modal contrastive learning [47]. Vokenization [68] proposes the issuall--supervise language model, which is closest to our work. Vokenization proposes to supervise language model to predict a visualized token, called 'voken' for each input text token. Vokens are obtained by contextualized token-to-image to-image matching model, pretrained on a MS COCO image captioning dataset [12]. In this work, we experiment with alternative objectives which avoid the approximation error from finite voken assignments. In addition, we use How NowToo100 T0100M [54] video dataset, which provides more diverse vocabulary as well as richer world commonsense and physical reasoning knowledge.
3
Video-Languag Knowledge Distillation
3.1 Method Overview
We aim to learn better language representation with the knowledge distilled from visual information. For this, we leverage two kinds of datasets: the aligned muti-modal dataset, Dv1.: {(x, v)} (e.g.. HowTo 100 [54]); and the text dataset, Dr.: {x) (e.g.. Wikipedia), where is sentence and is video paired with x. Our knowledge transfer is done in two stages: (1) cross-modal pretraining of a teacher model, M on multi-modal data DvL (Eq. 1) (2) distilling the knowledge of teacher model to a student model, MS, on text data D,. (Eq. 2). We illustrate our two-stage knowledge transfer method in Fig. 1.
min 8
E x.v
CT v)
min E KD 05 x-Dr
(MT M x))
(2)
Our teacher model M consists language model LM" and visual encoder V Both LM and TT have transformer [72] architectures, where LM" takes text tokens and V takes video frame
3

---

(a) Video-Language Contrastive Learning
(b) Masked Language Modeling
Triplet Margin Loss
Negativa Log-likelhhod
h
Average Pool
Poelive Put
h'
t
Teacher LM
Teacher LM
Teacher Visual Encoder
Teacher LM
+
+
t
0 Random caption
000
Video frame features
Paired caption
Masked Text mosked
Figure 2: Cross-modal pretraining of our teacher model on a multi-modal dataset (Sec. 3.2). We train our teacher model with (a) video-language contrastive learning and (b) masked language modeling. For video-language contrastive learning, we only illustrate the negative text samples for brevity.
features v as inputs. Our student model M a transformer language model LM sharing the same architecture with teacher language model LM illustrated Fig. ( we first train teacher models LM and V with contrastive learning and masked language modeling. Then, we distill the knowledge of teacher models to student model LM as in Fig. 1(b). In the following subsections, we discuss the detailed training procedure of teacher (Sec. 3.2, Fig. 2) and student models (Sec. 3.3, Fig. 3).
3.2 Teacher Model
We train our teacher model on a multi-modal dataset with two objectives, i.e., video-Language contrastive learning (Fig. 2(a)) and masked language modeling (Fig. 2(b)): L = LeT + LMLM"
Architecture. As shown in Figure 2, our teacher model R consists of a language encoder LM and visual encoder V" Both LM and V have similar transformer architecture." For each sen- tence x, we tokenize it and append special token [CLS] that represents the entire sentence following Devlin al. [19]. LM" takes x and outputs contextualized representation h" [h hf h" For each video v, we extract frame-level features e" with an off fff-the-sheff image encoder (see more details in Sec. 4.2). Note that the parameters of the image encoder are not updated to save computation. We feed the frame features e" {ef e%) to our visual encoder V to get contextualized video frame features h" (hf ... h We get the final video representation he by temporally averaging frame-level features: h" . E/' hy. Different from Tan and Bansal [68], both LM and V
parameters trained from scratch.
Video- Video-Language Contrastive Learning. To learn multi-mon modal grounding, we use contrastive learning objective that encourages closer distance between representations aligned video-texx pairs than unaligned pairs, as shown in Fig. 2 (a), For each x, we randomly sample another text x' from its batch with x' + x. Similarly, for each v, we randomly sample another video v' from its
In our experiments, different weights over the objectives not significantly change results. "In our experiments, we use the BERT architecture with two different configurations: 12 layers/768 hidden dimensions (BERT BERTHASE) and layers/512 hidden dimensions (BERTeus12u).

---

(a) Knowledge Distillation
(b) Masked Language Modeling
KD Objectives CRD)
Negativa Log-likellhood
+
t
0
0
0
O
0
t
t
Teacher Language Model (frozen)
Student Language Model
Student Language Model
Same Input Text
Masked Text
Figure 3: Illustration of our knowledge distillation from teacher language model LM to student language model LM on text dataset (Sec. 3.3). We train our student model with (a) knowledge distillation objectives and (b) masked language modeling.
batch with v' * v. Then, we use hinge loss max(0, a - pos neg) on cosine similarities:
Lcr(x,X
max cos (h h") + cos(h h"))
cos
(3)
+ max(0, a cos (h. h") + cos(h h" }
where a is the margin between the similarities a positive pair and a negative pair. Different from previous methods [7; 32] that exploit sentence-level contrastive loss, follow [68] to construct a loken-level contrastive loss (triplet margin loss) that grounds the visual information to each contextualized token output. This fine-grained contrastive loss will help the token-level knowledge distillation in Sec. 3.4.
Masked Language Modeling. For better language understanding in teacher model, follow BERT [19] to use masked language modeling (MLM) objective (Fig. 2(b)). By replacing 15% of tokens in with special token [MASK], obtain masked text x maskel with the same length. The model takes x masked as input and learns to predict the tokens by minimizing the negative log- EieMask log p(x; maskkod) ), where Mask refers to the indices
likelihoods: (MLM(X, masked of masked tokens.
3.3 Student Model
After train teacher model on a multi-modal dataset, transfer its knowledge student model on a text dataset. Following Kim and Rush [40], we train our student model with a sum of masked language modeling and two knowledge distillation objectives, NST and CRD (see Sec. 3,4):
LMLM
KD NST
KD LCRD
(4)
Architecture. As shown in Fig. 3 (a), our student model M is language model LM with the same transformer architecture the teacher language model T.M We train LMS from scratch. Following previous works [11: 26]. we introduce a multi-layer perceptron (MLP) distillation head on top of the last hidden states of LM In our ablation study in appendix, we find that adding a distillation head slightly improves the distillation performance.
3.4 Knowledge Distillation Objectives
We next describe the knowledge distillation (KD) objectives used in transferring knowledge from the teacher model LM (Sec. 3.2) to this student model LMS. Note that weights of the teacher model LMV" are frozen during the knowledge distillation process since the teacher model should not be affected by the student model's performance. Following Kim and Rush [40], use the knowledge
5

---

distillation objective combined with the MLM objective (Fig. 3 (b)). Concretely, we use the same input text mask for MLM and KD objectives. While we calculate the MLM loss only on masked positions, we calculate KD losses using all hidden states following Clark et al. [16] (Fig. 3).
We study the following KD objectives: Soft-label [33], 1.2 Regression [3], Neuron Selectivity Transfer (NST) [34], Contrastive Representation Distillation (CRD) [71], and Vokenization [68]. In our experiments comparing different KD objectives (Table 5), NST and CRD perform best, while the combination of them improved the performance even further. Therefore, we propose NST+CRD for our cross-modal knowledge distillation objective.
Soft Label: Hinton et al. [33] proposed a knowledge transfer method by taking a teacher model prediction with temperature scaling 'soft label'. We minimize cross-entropy between Pd (ylæ*) and pS (y)(z), i.e., the word output probabilities of LM and LM given the input text respectively:
X
lalt- (x)
pT (y:: 2/xx ps (y: 2|x)
(5)
where Z is the word vocabulary. Following Hinton et al. [33], we divide the softmax logits of LM" and LM by a temperature parameter T = 2.0. Note that for soft-label KD, we reuse the LM head, instead of learning an additional distillation head.
L2 Regression: Following Ba and Caruana [3] which uses feature regression for KD, we minimize the squared L2 distance between sxx and t(x), the last hidden states of LM" and LM given input text x:
x
KD Regressson (x)
[s(x); t(x)x1l
Neuron Selectivity Transfer (NST): NST [34] is KD method that transfers heatmap like spatial activation patterns of teacher neurons to student neurons. We transfer the sequential activation patterns of t(x) E x to s(x) E R xd where t(x) and s(x) are the last hidden states of LM" and L given input text x, and d is the hidden state dimension (# neurons). Following Huang and Wang [34], we use the squared maximum mean discrepancy (MMD) [25] with kernel trick to measure the distance between the activation patterns of student neurons {s(x),, Id and teacher neurons (t(x),., )/1=:
d
1
MMD*(x)
ke s(s)))
k (f(x)..;t(xx)
d
d²
i=1
#=1
d
d
2
k [s(()),,:t(x))) t(x)+a)
d
exp (- || 2
where use Gaussian kernel k|s; t]
1. We transfer the teacher activation patterns to the student by minimizing squared MMD: CKD (x) MMD"(x) (x)
with
Contrastive Representation Distillation (CRD): CRD [71] is a KD objective which maximizes the mutual information between the teacher and student representations with contrastive learning. Let's denote E S and t as student and teacher representations given x. We are given positive pair (drawn from the joint distribution) for every N (batch size) negative pairs (drawn from the product of marginals; independent randomly drawn inputs from and S). Following [71], we maximize the lower bound of mutual information between and by minimizing the following term:
CERD (x) (x (possivee) [log h(s, t)] N Bg(sst/reguiive) log h(s, t))]
(8)
Mss, t) exp (s) f2()))
exp (f1 f2())) N M
where M is the cardinality the dataset, f1, f2 are learned linear layers followed by 7.2 normalization, which map the student and teacher representations into same feature space. Since large N leads

---

to tight mutual information lower bound, following [71], we implement memory buffer that stores the latent features of each data sample computed from previous batches. Therefore, during training we can efficiently retrieve a large number of negative samples from the memory buffer. Note that since CRD is based on contrastive learning, the only KD objective where student and teacher language models can take different inputs.
Vokenization [68] could be viewed as a knowledge distillation method, where token- level text-to-image retrieval results (called "vokens") of a multi-modal matching model are used as labels for a student language model. For the i-th input token X,, we calculate cosine similarity between the i-th teacher language model hidden state t(x); and a video feature V. Out of 30K pre-selected videos, we select video that maximizes cosine similarity and use as the voken for x, By denoting the voken of x; as voken;, we formulate our vokenization-ber based KD objetive as:
Vokenization:
Vokon (x)
log voken (y: = voken; (x)
where PS Picken (y/x) is the voken classification probabilities of LM given input text x. We experiment with vokenization-basee KD by retrieving vokens from images and videos (see Table 6 of Sec. 5.2). Note that vokenization suffers from approximation error; it's hard cover diverse textual concepts with 30K vokens. This motivates us to experiment with different 'soft' KD objectives described in this section (see Table of Sec. 5.2).
4 Experimental Setup
4.1 Datasets
Video-Text Text Dataset. We use HowTo100M [54] for cross-modal pretraining of our teacher model
(Sec. 3.2). How 1o000000M has 1.22M videos totaling 136M video clips with total duration of 134,472 hours describing over 23K different visual tasks. There are 138M captions, 568M tokens with 633K distinct tokens.
Text Pretraining Dataset. To transfer the knowledge from our teacher language models to student language models (Sec. 3.3), we follow Tan and Bansal [68] use English Wikipedia. For ablation studies (Sec. 5.2), we use Wikil03 [52], widely used subset of English Wikipedia. There are 2.9B tokens and 120M sentences English Wikipedia, and I1M tokens and 4.2M sentences in Wikil03.
Text Downstream Dataset. Following Tan and Bansal [68], we finetune our models on GLUE [73], SQuAD [61] 1.0 and SQuAD2.0 [60], and SWAG [79] to assess the pretraining performance. Since some smaller tasks in GLUE are reported as unstable in recent papers [21], we evaluate on the
four largest datasets of GLUE: SST-2 [9], QNLI [61], QQP [35], and MNLI [74]. In addition,
also evaluate our models on the GLUE diagnostics [73], PIQA [6], and TRACIE [82] to measure its linguistic knowledge, physical reasoning, and temporal reasoning abilities.
4.2 Video Feature Representations
Following Miech et al. [54], we encode video features by concatenating features from a 2D frame- level image encoder and a 3D video encoder in channel dimension. Note that the parameters for 2D image encoder and 3D video encoder are not updated.
For the 2D image encoder, we sample video frames by 1fps (frame/second), The 2D image encoder outputs features for each frame individually. We experiment with GesNet-112 [30] pretrained on ImageNet- IK [18] and CLIP [58] image encoder (ViT-B/32 [23]). In contrast to conventional image encoders trained with image label classification, the CLIP image encoder is trained to match corresponding natural language description by large-scale contrastive learning. We discuss if this natural language supervision can help our cross-modal KD in Sec. 5.1.
For the 3D video encoder, we use 3D-ResNNeXt-152 [75; 28; 36] trained from a combination of publicly available datasets: ActivityNet [8], Kinetics [37], UCF-101 [65], and HMDB-51 [43]. The 3D video encoder processes 24fps videos with 3D convolution and yields features at 1.5fps. Then we sub-sample the features to lfps match the frame rate of 2D image encoder.
"https://gitte wtt.aa/.ccc.eeccaaaaaa///a/aa /DD Rewtless-PyToettt
7

---

Table I: Cross-modde knowledge distillation results of BERT 12L7768 student language model 7 downstream NLU tasks. the first block, include the image- basee-baeddd- vokenization (Img- Voken) and its text-only pretrained baseline performance from Tan and Bansal [68]. In the second block, we compare our cross-modal KD method (NST+CRD) to video-based vokenization Vid-Voken) and
text-only pretrained baseline. fEM refers to 'xxact Match'.
SST-2
QNLI
QQP
MNLI
SQuAD v1.1
SQuAD v2.0
SWAG
Avg.
Acc
Acc
Acc
Acc
EM
EM
Acc
BERT 12L7665 [68]
89.3
87.9
83,2
79,4
77.0
67.7
65.7
78.6
+ KD (Ime-Voken) [68]
92.2
88.6
88.6
82.6
78.8
68.1
70.6
81.4
BERT
89.0
88.0
86.2
79.2
77.2
68.0
65.0
78.9
+ KD (Vid- VokeV) w/ ResNet
93.4
89.2
88.7
83.0
78.9
68.7
70.0
81.7
+ (Vid- Voken) CLIP
94.1
89.0
83.9
79.2
68.6
71.6
82.3
(NST+CRD) w/ ResNet
94.2
89.3
89.7
84.0
79.0
68.9
71.8
82.4
KD (NST+CRD) w/ CLIP
94.5
89.6
89.8
84.2
79.6
68.7
72.0
82.6
4.3
Implementation Details
For the student distillation head, we use a two-layer MLP with ReLU activation. For both student and teacher language models, following previous works [49; 17; 68], truncate input text that is longer than 128 tokens. We truncate videos features that are longer than 512 frames. We use an AdamW [41] optimizer with learning rate 2e-4 and weight decay [50] of 0.01. We reserve 10K samples of the How 100To000M dataset as validation data. We train the teacher model until it converges on validation data. For downstream tasks, we report the results on the validation sets. We train epochs with a learning rate of le-4 and a batch-size of 32 for all downstream tasks. We use hinge loss margin 1.0 for LeT (Eq. 3). We implement our models with PyTorch 1.5 [56] and train them with Nvidia GeForce RTX 2080ti GPUs. For teacher pretraining, we use 4 GPUs for BERT 121 and BERT 61112 models for days and 2.5 days respectively. For knowledge distillation, we use GPUs for BERT and BERT models for 10 days and 3 days respectively.
5
Results and Analysis
5.1 Primary Downstream Task Results
In the first block of Table 1, we include the image-based vokenization Vokg-Voke)) and their text-only pretrained baseline from Tan and Bansal [68]. Given our reproduced text-only baseline shows a similar average performance (78.9 vs 78.6), our student models distilled from NST+CRD much better (82.6 vs 81.4). We discuss the comparison between video- based and image-based KD in detail in the following ablation study in comparison to vokenization (Table 6).
In the second block of Table 1, we compare our proposed cross-modal KD method (NST+CRD) to vidco-based vokenization (Vid- (okeVo and non-KD baseline (BERT (2L781)) which is only pretrained on text. We can see both cross- modal KD methods (i.c., KD and Vid-Voken) significantly outperform the text-only baseline across all 7 downstream tasks. We also experiment with different 2D frame encoders (Sec. 4.2): ResNet and CLIP. For both Vid- Voken and NST+CRD, we observe CLIP further improves the performance results over ResNet, indicating using strong visual encoder helps the teacher training and thus benefits the knowledge distillation.
5.2 Ablation Studies
In this section, we conduct a comprehensive ablation study to show the effectiveness of our proposed methods. For all ablation experiments, we use BERT architecture for student and teacher language models. We use ResNet-152 for 2D frame encoder and 3D-ResNeXt-155 for 3D frame encoder (Sec. 4.2). Wikil03 [52] is used for student model training. We also perform ablation experiments on the effect of additional distillation head appendix.
'Vokenization uses pretrained BERT checkpoint for its 'teacher' (vokenizer) model but we train our teacher language model fully from scratch.

---

Text-only Pretraining. Our cross- modal KD improves the performance on downstream NLU tasks significantly (Sec. 5.1). Where does the improvement come from, video or text? To answer this question, we conduct text-only pre- training of BERT 2H on Wikil03 text (11IM tokens), HowTo 100M captions (568M tokens) and compare them to no-pretrain baseline. In Table 2, while both pretrained models improve the performance over the no-pretrain baseline, Wiki 103-trained model outperforms HowTo To100M-trained model (which has more tokens) significantly. This indicates that our KD methods improve NLU performance because of multimodal grounding, instead of just the larger corpus.
Table 2: Text-only pretraining results of 6L/11HH pretrained on Wikil03, HowTo100M captions, and no- pretrain baseline.
Protrained on
SST-2
QNLI
QQP
MNLI
No- No-Pretrain
79.6
61.5
72.7
61.6
Wikil03 (Formal language)
88.8
84,9
85,3
77.4
HowTo 1000M (ASR captions)
83.3
78.5
83.7
71.5
Effect of Teacher Training Objectives. We here analyze the teacher training objec- tives by comparing the corresponding dis- tilled student model results. In Table the teacher model trained solely with MLM (+KD from T.M.M does not significantly change the student model performance. At the same time, the teacher model trained with only visual supervision, i.e., contrastive objective (+KD from TCT). improves the result. This illustrates the motivation to perform knowledge transfer from visually supervised MLM model. Lastly, combining the MLM and the contrastive objective (+KD from T.M.M+ in teacher model training shows the best student results.
Table 3: Ablation results showing the effect of the teacher model's ' training objectives. NST is used for knowledge distillation.
SST-2
QNLI
QQP
MNLI
BERT
88.8
84.9
85.3
77.4
+KD from TALM
88.1
83.1
85.6
77.4
+DD from T
88.9
85.2
86.2
77.5
+KD from 2 CT
91.1
85.0
87.4
78.4
Two-stage PT vs. Cross-modal KD. In Table 4, we compare two-stage pretraining with a single model to our proposed cross- modal KD approach. For single model base- lines, we use text-only (MLM on Wikil03), video-only (MLM+CT and two-stage (video-ther ttalo--then-tet)) pretraining. While the two-stage pretraining shows better results than the text/video-only pretraining,
Table 4: Comparison of pretraining on text, video, both (Two-stage PT), and our VidLanKD.
Model
SST-2
QNLI
QQP
MNLI
Text Video PT
88.8
84.9
85.3
77.4
HowTo100M1
84.0
78.9
84.2
73.1
Two-Stage
90.3
85.0
87.2
76.9
VIDLANKI
91.1
85.0
87.4
78.4
VIDLANKD outperforms all baselines GLUE tasks, especially on SST-2 and MNLI.
KD Objectives Comparison. In Table 5, compare different knowledge distillation objectives introduced in Sec. 3.4. The stu- dent models trained with NST [34] and CRD [71] show the best finetuning performance on downstream tasks. When combining NST and CRD, performance further improves with marginal additional computation cost, hence we propose to use NST+CRD for our rrosll knowledge distillation.
Table 5: Ablation of knowledge distillation objectives.
SST-2
QNLI
QQP
MNLI
BERTuUII2
88.8
84,9
85.3
77.4
+KD Soft label
87.2
84.4
86.4
76.6
*KD-Regression
88.8
87.1
78.1
+KD- Vid Voken
89.7
85.5
86.5
77.8
+KD-NST
91.1
85.0
87.4
78.4
+KD-CRD
90.0
85.5
87.3
78.3
+KD-NST-CRD
91.5
85.8
87.4
78.7
Comparison to Vokenization. In Table 6, we compare NST [34] and Vokenization [68] in both image and video-level teacher model supervision. For video-level supervision, we provide our visual encoder with the whole video features (Sec. 4.2). For image-level supervision, we provide our visual encoder only with 2D features of the middle frame for each video clip. With image-level super-
Table 6: Comparison between vokenization (Voken) and NST with image and video-level supervision.
SST-2
QNLI
QQP
MNLI
BERT BERT&UIIE GUS1ZH
88.8
84.9
85.3
77.4
+KD-Voken (Image)
89.3
84.4
86.0
77.5
+DD-NST (Image)
88.9
85.0
86.3
77.2
+DD DD-Veeen (Video)
89.7
85,5
86.5
77.8
KD-NST (Video)
91.1
85.0
87.4
78.4

---

vision (first block), Vokenization and NST show comparable performance. However, with video-level supervision (second block), NST outperforms Vokenization on 3 out of 4 tasks. The gap in the video domain might come from voken approximation error, where each image or video input is approximated with one of 30K predefined vokens. Since videos usually contain more diverse contents than images, the voken approximation error would be amplified in video-level supervision, whereas our NST distillation avoids this issue.
5.3 Analyzing the Knowledge Learned from Video
In this subsection, we analyze the knowledge that our language models Iearn from video via cross- modal knowledge distillation. To measure linguistic knowledge and physical/teporral reasoning ability, we show results of our models on the GLUE diagnostics [73], the Physical Interaction Question Answering (PIQA) [6], and TRACIE [82]. In addition, we visualize the learned multi- modal grounding ability of our model with text-to-video retrieval.
Table 7: Finetuning performance on GLUE diagnostics [73]. PIQA [6] and TRACIE [82] datasets, which measure the linguistic knowledge, physical and temporal reasoning capabilities of language models, respectively.
GLUE diagnostics
PIQA
TRACIE
Lexicon
Predicate
Logic
Knowledge
BERT 6LS12H
53.0
64.2
44.5
44.0
56.9
63.4 66.7 (+3.3)
KD-NST
53.3 (+0.3)
63.7 (-0.5)
44.8 (+0.3)
48.6 (+4.6)
60.0 (+3.1)
Linguistic Knowledge. GLUE diagnostics dataset [73] evaluates sentence understanding through natural language inference (NLI) problems. The dataset consists of sentence pairs labeled with their entailment relations (entailment, contradiction, neutral) in both directions and tagged with set of entailment labels. Each example in the dataset is labeled with 4 categories of linguistic phenomena: (1) lexical semantics, (2) anddicate-argument structure, (3) logic, and (4) knowledge (including common sense). In Table 7, we compare the baseline language model (BERT 6L/S12 pretrained on Wikil03) to our NST-distilled model. We finetune the models on MNLI [74] that has the same format and test on GLUE diagnostics. We observe large gain on the knowledge category (which involves common sense and external world knowledge) while there no significant differences on other categories. This suggests that our student model learns the external, grounded world knowledge in the teacher model and the video-text dataset.
Physical and Temporal Reasoning. PIQA [6] is question answering dataset evaluating physical interactions and commonsense reasoning. TRACIE [82] is a temporal reasoning benchmark on implicit events, which are not mentioned explicitly natural language text but can be inferred from it. In Table 7, our BERT distilled with NST significantly outperform the text-only pretrained baseline on both benchmarks. The finding suggests (consistent with the GLUE diagnostics findings above) that video knowledge distillation also helps improve the physical and temporal reasoning capabilities of the language model. See appendix for the more detailed discussion on the PIQA and TRACIE experiment.
Conclusion
We introduce VIDLANKD, a novel cross-modal knowledge distillation method to help general language understanding. Our teacher model first trained on a video-text dataset, and then we transfer its knowledge to student language model with text dataset. Via the distillation objectives and video-text datasets, our method overcomes the limitations of the recent vokenization method. We empirically demonstrate that our VIDLANKD improves on several NLU tasks over models trained by pure-text or vokenization. We conduct comprehensive ablation analysis to show the effectiveness of each proposed component. We also illustrate the linguistic knowledge and physical/temporal commonsense reasoning learned from videos, and visualize our model's 's multi-mode grounding ability.
10

---

Acknowledgments
We thank the reviewers for their helpful comments. We thank Yixin Nie and Gabriel Ilharro for useful dataset suggestions. This work was supported by ARO-YIP Award W91 191-NN-18-0 DARPA MCS Grant N66001-19-2-400.
DARPA KAIROS Grant FA8750-19 1-:00-19---110- 2-0-4044, Google Focused Research Award, and Bloomberg Data Science Ph.D. Fellowship. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
References
[1] Rohan Anil. Gabriel Pereyra. Alexandre Passos, Robert Ormandi, George Dahl, Geoffrey Hinton. 2018. Large scale distributed neural network training through online distillation. arXiv preprint arXiv:1804.0333
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV.
[3] Lei Jimmy Ba and Rich Caruana. 2014. deep nets really need to deep? In NeurIPS.
[4] Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understand- ing the age of data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3185-5198, Online. Association for Computational Linguisties.
[5] Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et 2020. Experience grounds language. In EMNLP.
[6] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI, pages 4432-7439.
Patrick Bordes, Eloi Zablocki, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. 2019. Incorpo- rating visual semantics into sentence representations within a grounded space. EMNLP-IJCNLP.
[8] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: large-scale video benchmark human activity understanding. CVPR, pages 961-970.
[9] Daniel Cer, Mona Diab, Encko Agire,, Inigo Lopez-Gazpio and Lucia Specia. 2017. Semeval-2011 task 1: Semantic textual multtiiiiammliiliaaallaa and cross loossll focused evaluation. In SemEval.
[10] Yevgen Chebotar and Austin Waters. 2016. Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech, pages 3439-3443.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi. and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR.
[12] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C LLwrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 104.00550400032.
[13] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed. Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Learning universal image text text representations. In ECCV.
[14] Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. 2018. Darkrank: Accelcrating deep metric learning via cross sample similarities transfer. In AAAI.
[15] Gordon Christie, Ankit Laddha, Aishwarya Agrawal, Stanislaw Antol, Yash Goyal, Kevin Kochersberger, and Dhruv Batra. 2016. Resolving language and vision ambiguities together: Joint segmentation & prepositional attachment resolution captioned scenes. In EMNLP.
[16] Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR.
[17] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave. Myle Ott. Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR.
11

---

[19] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
[20] Tuong Do, Thanh-Toan Do, Huy Tran, Erman Tjiputra, and Quang D Tran. 2019. Compact trilinear interaction for visual question answering. In ICCV, pages 392-401.
[21] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint or Xiv: 06305.
[22] Li Dong, Nan Yang. Wenhui Wang. Furu Wei. Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wure Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NewrIPS.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR.
[24] David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003, English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34.
[25] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012.
A kernel two-sample test. JMI.R, 13(25):72--773
[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre HI Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. 2020. Bootstrap your own latent: A new approach to self superviue learning. In NeurlPS.
[27] Saurabh Gupta, Judy Hoffman, and Jitendra Malik. 2016. Cross modal distillation for supervision transfer. In CVPR, pages 2827-2836.
[28] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. 2018. Can spatiotemporal 3d cans retrace the history of 2d cnns and imagenet? In CVPR.
[29] Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1))333--34 346.66......
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR, pages 770-778.
[31] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Jumyuan Xie, and Mu Li. 2019. Bag of tricks for image classification with convolutional neural networks. In CVPR, pages 558-567.
[32] Jack Hessel, Lillian Lee, and David Mimno. 2019. Unsupervised discovery of multimodal links in multi-image, multi-sentence documents. In EMNLP-IJCNIPP pages 2034-2045.
[33] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. In NewrIPS Deep Learning Workshop.
[34] Zehao Huang and Naiyan Wang. 2017. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: 1707.01219.
[35] Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs.
[36] Hirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh. 2020. Would mega-scale datasets further enhance spatiotemporal 3d cans? arXiv preprint arXiv:2004.0490
[37] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv: 000..000559995
[38] Douwe Kiela, Alexis Conneau, Allan Jabri, and Maximilian Nickel. 2018. Learning visually grounded sentence representations. In NAACL.
[39] Douwe Kiela, Ivan Vulic, and Stephen Clark. 2015. Visual bilingual lexicon induction with transferred convnet features. In EMNLP. ACL; East Stroudsburg, PA.
[40] Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In ACL.
[41] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR.
12

---

[42] Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler. 2014. What are you talking about? texi-lo-image coreference. In CVPR. pages 3558-3565.
[43] Hildegard Kuchne, Hueihan Jhuang. Estibaliz Garrote, Tomaso Poggio, and Thomas Serr. 2011. Hmdb: large video database for human motion recognition. In ICCV, pages 2556 2563. IEEE.
[44] Zhenzhong Lan, Mingda Chen, Schastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. ICLR.
[45] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou, 2020. Unicoder-vl: universal encoder for vision and language by cross-modal pre-training. In AAAI, pages 1 11336-11344
[46] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical encoder for video+ language onmi-representation pre-training. In EMNLP.
[47] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2020. Unimo: Towards unitied- modal understanding and generation via cross cross-modal contrastive learning. arXiv preprint arXiv:20/2. 15409.
[48] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. ECCV, pages 740-755. Springer.
[49] Yinhan Liu,, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:
/ /907. 1692.
[50] Iyya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In ICLR.
[51] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for rision-and-language tasks. In Neur/PS.
[52] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR.
[53] Antoine Miech, Jean- nan-Baptiste Alayrae, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. In CVPR.
[54] Antoine Micch, Dimitri Zhukov, Jean- can--Batiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2630-2640.
[55] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In AAAI.
[56] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer, 2017, Automatic differentiation in Py PoTch, In NIPS Autodiff Workshop.
[57] Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint 211:22033000206
[59] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
[60] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018, Know what you don't know: Unanswerahle questions for squad. In ACI..
[61] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP.
[62] Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015, A neural attention model for abstractive sentence summarization. In EMNLP.
13

---

[63] Abigail See. Peter Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL.
[64] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In ICML.
[65] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Uef101: A dataset of 101 human actions classes from videos in the wild. CoRR.
[66] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019, Videobert: A joint model for video and language representation learning. In ICCV.
[67] Hao Tan and Mohit Bansal, 2019. Lxmert: transformers. In EMNLP.
Learning cross-modality encoder representations from
[68] Hao Tan and Mohit Bansal. 2020. Vokenization: improving language understanding with contextualized, visual-grounded supervision. In EMNLP.
[69] Jiaxi Tang and Ke Wang. 2018. Ranking distillation: Learning compact ranking models with high performance for recommender system. In SIGKDD, pages 2289-2298.
[70] Zineng Tang. Jie Lei, and Mohit Bansal. 2021. Decembert: Learning from noisy instructional videos via dense captions and entropy minimization. In NAACL-HLT, pages 2415-2426.
[71] Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive representation distillation. In ICLR.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NewrIPS.
[73] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, 2018, Glue: A multi-task benehmark and analysis platform for natural language understanding. In ICLR.
[74] Adina Williams. Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.
[75] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, 2018. Rethinking spatiotemporal feature learning: aeceracc tradc-off's in video classification. In ECCV.
[76] Jun Xu, Tao Mei. Ting Yao. and Yong Rui. 2016. Msr-vit: A large video description dataset for bridging video and language. In CVPR.
[77] Zheng Xu, Yen- Chang Hsu, and Jiawei Huang. 2018 Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks. In ICLR Workshop.
[78] Zhilin Yang. Zihang Dai. Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xinet: Generalized autoregressive pretraining for language understanding. In NewrIPS.
[79] Rowan Zellers. Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP.
[80] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In CVPR, pages 4320 4328.
[81] Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eichiro Sumita, Zuchao Li, and Hai Zhao. 2019. Neural machine translation with universal visual representation. In ICLR.
[82] Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1361-1371, Online. Association for Computational Linguistics.
[83] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision- languaee pre- prettrining for image captioning and vqa. In AAAI.
[84] Luowei Zhou, Chenliang Xu, and Jason J Corso. 2017. Towards automatic learning of procedures from web instructional videos. In AAAI.
[85] Linchao Zhu and Yi Yang. 2020. Acthert: Learning global-local video-text representations. In CVPR.
14


Reviews:
[{"review_id": "oWp7GN4KYsL", "reviewer": "Reviewer_X96L", "summary": "The paper proposes to improve language understanding using a video-language knowledge distillation method. Compared to the previous state of the art approach, “Vokenization”, they use a large multi-modal video dataset for video-language pre-training to accommodate larger vocabulary needed for better language understanding. For their knowledge distillation based approach, the teacher is the pre-trained model on the multi-modal dataset whose weights are distilled to the student language model trained only on a text dataset. This also avoids the approximation error in the vokenization approach. They perform extensive experiments and improve performance on several downstream language understanding benchmarks (GLUE, SQuAD and SWAG) and also show the commonsense knowledge and reasoning learned from videos. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is marginally novel and looks at an interesting side of improving language understanding models using video-language based pre-training. It is well written and comprehensive. They perform extensive experiments to show the effectiveness of their method on downstream tasks and several ablation studies to compare different alternatives for the loss functions and pre-training datasets.\n\n1. In line 104, the authors discuss one of the limitations of the vokenization paper - the approximation error and that their method addresses this. It is though not very clear what this error is and how the method takes care of it? \n\n2. Vokenization trains on MSCOCO which is a relatively small image-text dataset, but combining it with Conceptual Captions [1] or SBU captions [2] datasets wouldn’t be enough? What is training on video datasets bringing extra than just on the combination of the above image datasets?\n\n3. What happens if we take the teacher language model after cross-modal pretraining and fine-tune on the text pre-training dataset without knowledge distillation? I believe this is different from the baseline mentioned in Table 1., first row.\n\n4. Why are the ablation experiments done on BERT 6L/512H whereas the main model is trained on BERT 12L/768H ?\n\n5. Although the models show effectiveness over different tasks and datasets, the novelty of the method is limited as it uses state of the art knowledge distillation and pretext self-supervised objectives for training the model. \n\n[1]Sharma, P., N. Ding, S. Goodman, et al. Conceptual captions: A cleaned, hypernym, image-text dataset for automatic image captioning, ACL, pages 2556–2565. 2018\n[2] Ordonez, V., G. Kulkarni, T. L. Berg. Im2text: Describing images using 1 million captioned photographs, NIPS 2011 \n\n\n\n##Post Rebuttal\n\nI have carefully read the authors' response and they address my concerns. I appreciate the new experimental results provided by the authors and they give a better idea of the issue raised. I maintain my original rating.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I recommend the authors to add the broader impact and limitations of the paper from the supplementary to the main paper. The main limitation discussed by the authors is the bias in the video datasets (gender, racial, etc.) and this bias getting amplified during training. \nAnother potential limitation is the carbon footprint and cost effectiveness of training on these large-scale video datasets. Is there any alternative where you could still use visual information for improving language understanding in a more environmentally friendly way ? An ideal proposition but worth thinking about. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "knNF1R9w276", "reviewer": "Reviewer_hPyo", "summary": "The paper proposes a new method to improve language understanding with the help of the vision modality. In the proposed method VidLanKD, a teacher model is firstly pre-trained with video-language contrastive loss on Howto100M data. Then the LM part of the teacher model is used to distill knowledge to a student LM model.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using vision modality to help language understanding definitely has huge research value. However, I find the paper lacks detailed explanations on some design choices and also on the reasons why it is more preferable compared with Vokenizer.\n\n1. Why do we need a distillation process from the teacher LM to another model? What will happen if we directly pre-train more with only text on the teacher LM pre-trained with image-caption pairs, and fine-tune it on downstream tasks?\n\n2. The paper mentions one limitation of vokenization, images cannot represent words well that require more activity-based commonsense, which is valid. However, how can the proposed method address this problem?\n\n3. Both the teacher model pre-training and the distillation have multiple losses. I feel that the ratio of those losses is a very important hyper-parameter that can significantly affect the performance. For example, for the teacher LM pre-training, since the captions are simply too short and few compared with the unsupervised text, if we don’t set a high coefficient for the contrastive side,  the MLM loss will dominate, which would make the whole learning-from-vision idea pointless. While the paper lacks thorough discussions on this.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "60Uy0Qd3TXl", "reviewer": "Reviewer_ZhJy", "summary": "The paper studies the problem of how to use visual information to help NLP models. It argues that videos contain rich visual grounding information and could potentially benefit pure-text NLP tasks. It proposes to distill such visual information into a language model during pre-training. Experiments on GLUE benchmark suggest the efficacy of the method.\n\nOverall, I applaud the novelty and problem the paper tries the address. However, the experiments could be strengthened by better control of the baselines and more analysis (see cons). I lean towards acceptance but I would suggest adding the suggested experiments later if possible.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\nThe paper tackles a long-standing problem: grounding NLP models. The paper takes a practical yet interesting angle: distill grounding information in the pre-trained language models so it could benefit various NLP tasks.\n\nThe distill approaches make sense and works well in experiments.\n\n\n\nCon:\n\t\n1. Potentially important baselines missing. To verify that the gain is from injected visual information, the authors introduces a student model that is pre-trained on HowTo100M. However, another possible contributing factor is the distillation process, as the distillation process itself might bring performance improvement, as long as the teacher is a reasonable model. \nThus, I think a baseline where the teacher is trained only on the text of HowTo100M/Wiki and distilled to a student should be included to disentangle the effect of the visual information and the effect of the distillation process.\n\n2. Concern over the setting. Is the teacher model initialized from BERT? If so, then a baseline where the teacher is simply the original BERT should be included, to investigate if some of the useful information distilled into the student is from BERT.\n\n3. Lack of analysis. While showing improvement on GLUE benchmarks is a good sign, it is still hard to see how the grounding information helps a concrete NLP task (e.g., it might be hard to imagine how exactly videos help SST-2). Some examples of how the grounding helps PIQA or Knowledge test in GLUE could be helpful.\nAnother potential aspect I could think of is to test whether the grounded pre-training relieves reporting bias [1,2].\nI won’t count this as a reason for rejection but would appreciate a better analysis.\n[1] Reporting bias and knowledge acquisition. AKBC 2013.\n[2] Do Neural Language Models Overcome Reporting Bias? COLING 2020.\n\t", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing I could think of.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4H2HelNLUA", "reviewer": "Reviewer_aMi4", "summary": "The paper proposed to distill the large-scale pre-trained video-language multi-modal knowledge into the language modeling. Different knowledge distillation objectives are applied. In the downstream task, the distilled language model shows performance improvement over the conventional language model. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The novelty comes from dIstilling the video-language multimodal knowledge learned by video-text pre-training into the language modeling. That could enable the language model the ability to deal with more visual-relevant commonsense such as action.\n+ Multiple knowledge distillation methods are well studied and compared. \n+ Proposed method outperforms the conventional language model pre-trained only by text in quite a few downstream tasks.\n+ The fair comparison against the Vokenization method further validates the effectiveness of the proposed framework. \n\nWeakness:\n- When comparing with Vokenization, 30K videos are pre-selected according to L205. A question about the detail is how to select the videos?\n- In the downstream tasks, it is interesting to see in what samples your model is generally better than text-pretrained language model.  Does that correspond to your motivation that your model should help in activity-based and physical commonsense knowledge?\n\nClarity:\nThe paper is clearly illustrated and well written. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All but one reviewer recommend the acceptance of this paper.

The reviewer not recommending acceptance has remaining concerns about positioning this work with respect to vtokenizer and the AC encourages the authors to address this. Please also be careful about explaining additional details in this submission as requested by this reviewer. However other reviewers were quite positive about this work and one reviewers noted that they felt that this paper does indeed make a more convincing case for the general topic of creating visually-grounded NLP models than Vokenization.

The AC recommends acceptance.

Decision: Accept (Poster)