VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer

1 Introduction . Language learning can be aided by grounded visual cues , as they provide powerful signals for modeling a vastness of experiences in the world that can not be documented by text alone [ 5 ; 29 ; 4 ] . While the recent trend of large-scale language model pretraining indirectly provides some world knowledge from text , most large text corpora ( e.g. , Wikipedia ) do not provide enough multi-modal grounding information . Previous works have explored multiple ways of grounding language to visual information such as constructing a common vector space [ 38 ; 7 ] and supervising the model with token-wise generated vision labels [ 68 ] . However , the widely-used image-text datasets ( e.g. , MS COCO [ 48 ] ) are much smaller than text-only corpora in terms of word counts and vocabulary diversity for language learning . The recent method of ‘ vokenization ’ [ 68 ] is a promising initial step towards addressing this problem by supervising language models with weakly-aligned vision-language groundings . Firstly , an imagetext matching model retrieves a corresponding image to each text token in a sentence . Then a language model learns to predict the selected image ( called ‘ voken ’ ) for each text token . This can be seen as a knowledge distillation ( KD ) process [ 33 ] from a vision-language grounding model to a language 1Code and models : https : //github.com/zinengtang/VidLanKD 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . model . Although the voken classification task helps the language model to improve on natural language understanding ( NLU ) tasks , there exist several limitations : ( 1 ) images can not faithfully convey word meanings that require more activity-based and physical commonsense knowledge . ( 2 ) the voken supervision suffers from approximation/quantization error of the text-to-image retrieval . To address these problems , we propose a novel Video-and-Language Knowledge Distillation method , named VIDLANKD . Our teacher model consists of a video encoder and a language encoder . They are jointly trained with a video-language contrastive learning objective and a masked language modeling ( MLM ) objective on a multi-modal dataset ( see Fig . 1 ) . Then , we transfer the knowledge of the frozen teacher language encoder to a student language model by minimizing the distance between contextualized text representations of two models on a text dataset . For this , we propose to use different KD objectives including neuron selectivity transfer ( NST ) [ 34 ] and contrastive representation distillation ( CRD ) [ 71 ] that avoid the approximation error from voken assignments [ 68 ] . For cross-modal pretraining of our teacher model , we use HowTo100M [ 54 ] , a large-scale video dataset which has more diverse vocabulary and richer world commonsense ( e.g. , physical and temporal ) knowledge compared to MS COCO image dataset . In our experiments , student language models learned with the proposed video-language KD objectives outperform the baseline text-pretrained language models and the models distilled with vokenization , on several diverse natural language understanding benchmarks including GLUE [ 73 ] , SQuAD [ 61 ] , and SWAG [ 79 ] . We also show comprehensive ablation studies on video encoders , student KD objectives , teacher pretraining objectives , and video vs. image-based pretraining . Furthermore , we empirically illustrate that our model successfully learns linguistic world knowledge and physical/temporal commonsense abilities from video , by showing improved performances on the GLUE-diagnostics [ 73 ] , PIQA [ 6 ] , and TRACIE [ 82 ] datasets . Overall , our contributions are : ( 1 ) a novel cross-modal knowledge distillation method for improving natural language understanding , ( 2 ) using rich video-text data which can overcome the limitations of image vokenization , ( 3 ) empirical improvements on several language understanding benchmarks and studying different knowledge distillation methods , and ( 4 ) analysis on linguistic/physical/temporal knowledge learned from videos and ablation studies on the effectiveness of proposed components . 2 Related Work . 2.1 Knowledge Distillation . Knowledge distillation ( KD ) [ 33 ] is the process of transferring knowledge from a teacher model to a student model . It has been successfully used in a wide range of applications , such as machine translation [ 40 ] , visual recognition [ 31 ] , speech recognition [ 10 ] , and recommendation systems [ 69 ] . Recent works advanced the field of knowledge distillation by proposing new architectures [ 77 ; 80 ; 1 ; 55 ] and objectives [ 34 ; 14 ] . While many KD works study the problem of knowledge transfer within the same modality , crossmodal knowledge distillation [ 27 ; 20 ; 71 ] tackles the knowledge transfer across different modalities . Gupta et al . [ 27 ] transfers the knowledge of a model trained with RGB images to another model for depth maps and optical flow . Do et al . [ 20 ] proposes a KD method for visual question answering [ 2 ] , where the trilinear ( image-question-answer ) relational representation of a teacher model is transferred to a bilinear ( image-question ) student model . Tian et al . [ 71 ] combines contrastive learning and knowledge distillation to improve the knowledge transfer between different modalities . Our VIDLANKD transfers the knowledge of a multi-modal teacher model learned from a video dataset to a student language model that tackles natural language understanding tasks . 2.2 Language Pretraining . Large-scale pretraining of contextualized language models has seen huge success in natural language processing in recent years . ELMo [ 57 ] proposes to pretrain and fine-tune a large recurrent language model , which improves performance on a diverse set of downstream natural language processing tasks . BERT [ 19 ] improves the scalability of the pretrain/fine-tune framework by using a transformer [ 72 ] language model with a masked language modeling objective . Since then , pretraining of transformer language models has been extensively explored [ 49 ; 78 ; 44 ; 22 ; 64 ; 59 ; 16 ] for various natural language understanding [ 61 ; 74 ; 79 ; 73 ] and generation tasks [ 24 ; 62 ; 61 ; 63 ] . 2.3 Multi-modal Pretraining . Following the success of language pretraining with transformer models , pretraining of image-text [ 67 ; 51 ; 13 ; 47 ; 83 ; 45 ] and video-text [ 66 ; 54 ; 85 ; 53 ; 46 ; 70 ] multi-modal transformers have achieved improvements on numerous multi-modal downstream tasks [ 2 ; 76 ; 84 ] . The multi-modal transformers take both visual and textual inputs and are pretrained on image-text or video-text pairs with multimodal masked language modeling objectives . Despite the success on multi-modal downstream tasks , Tan and Bansal [ 68 ] finds that the multi-modal pretraining does not improve ( and sometimes even harms ) the language understanding performance . This is because the scale and diversity of text vocabulary of image-text and video-text datasets are usually smaller than those of text datasets . To utilize the rich vocabulary of text dataset , our VIDLANKD transfers the knowledge of pretrained multi-modal model to a student language model with a large text dataset . 2.4 Visually-Grounded Language Learning . A series of works explore using visual information to aid language understanding and generation tasks including co-reference resolution [ 42 ; 15 ] , machine translation [ 81 ] , bilingual lexicon learning [ 39 ] , and multi-modal contrastive learning [ 47 ] . Vokenization [ 68 ] proposes the visually-supervised language model , which is closest to our work . Vokenization proposes to supervise a language model to predict a visualized token , called ‘ voken ’ for each input text token . Vokens are obtained by a contextualized token-to-image matching model , pretrained on a MS COCO image captioning dataset [ 12 ] . In this work , we experiment with alternative objectives which avoid the approximation error from finite voken assignments . In addition , we use HowTo100M [ 54 ] video dataset , which provides a more diverse vocabulary as well as richer world commonsense and physical reasoning knowledge . 3 Video-Language Knowledge Distillation . 3.1 Method Overview . We aim to learn a better language representation with the knowledge distilled from visual information . For this , we leverage two kinds of datasets : the aligned muti-modal dataset , DVL : { ( x , v ) } ( e.g. , HowTo100M [ 54 ] ) ; and the text dataset , DL : { x } ( e.g. , Wikipedia ) , where x is a sentence and v is a video paired with x . Our knowledge transfer is done in two stages : ( 1 ) cross-modal pretraining of a teacher model , MT , on multi-modal data DVL ( Eq . 1 ) ( 2 ) distilling the knowledge of teacher model to a student model , MS , on text data DL ( Eq . 2 ) . We illustrate our two-stage knowledge transfer method in Fig . 1. min θT E x , v∼DVL LT ( MT , x , v ) ( 1 ) min θS E x∼DL LKD ( MT , MS , x ) ( 2 ) Our teacher model MT consists a language model LMT and a visual encoder VT . Both LMT and VT have transformer [ 72 ] architectures , where LMT takes text tokens x and VT takes video frame features v as inputs . Our student model MS is a transformer language model LMS sharing the same architecture with teacher language model LMT . As illustrated in Fig . 1 ( a ) , we first train teacher models LMT and VT with contrastive learning and masked language modeling . Then , we distill the knowledge of teacher models to student model LMS as in Fig . 1 ( b ) . In the following subsections , we discuss the detailed training procedure of teacher ( Sec . 3.2 , Fig . 2 ) and student models ( Sec . 3.3 , Fig . 3 ) . 3.2 Teacher Model . We train our teacher model on a multi-modal dataset with two objectives , i.e. , video-language contrastive learning ( Fig . 2 ( a ) ) and masked language modeling ( Fig . 2 ( b ) ) : LT = LCT + LMLM 2 Architecture . As shown in Figure 2 , our teacher model MT consists of a language encoder LMT and a visual encoder VT . Both LMT and VT have similar transformer architecture.3 For each sentence x , we tokenize it and append a special token [ CLS ] that represents the entire sentence following Devlin et al . [ 19 ] . LMT takes x and outputs contextualized representation hx = { hx [ CLS ] , hx1 · · ·hx|x| } . For each video v , we extract frame-level features ev with an off-the-shelf image encoder ( see more details in Sec . 4.2 ) . Note that the parameters of the image encoder are not updated to save computation . We feed the frame features ev = { ev1 · · · ev|v| } to our visual encoder V T to get contextualized video frame features hv = { hv1 · · ·hv|v| } . We get the final video representation hv by temporally averaging frame-level features : hv = 1|v| ∑|v| i=1 h v i . Different from Tan and Bansal [ 68 ] , both LM T and VT parameters are trained from scratch . Video-Language Contrastive Learning . To learn multi-modal grounding , we use a contrastive learning objective that encourages a closer distance between representations of aligned video-text pairs than unaligned pairs , as shown in Fig . 2 ( a ) . For each x , we randomly sample another text x′ from its batch with x′ ̸= x . Similarly , for each v , we randomly sample another video v′ from its 2In our experiments , different weights over the objectives did not significantly change the results . 3In our experiments , we use the BERT architecture with two different configurations : 12 layers/768 hidden dimensions ( BERT12L/768H = BERTBASE ) and 6 layers/512 hidden dimensions ( BERT6L/512H ) . batch with v′ ̸= v. Then , we use hinge loss max { 0 , α− pos + neg } on cosine similarities : LCT ( x , x′ , v , v′ ) = |x|∑ i [ max { 0 , α− cos ( hxi , hv ) + cos ( hx ′ i , h v ) } ( 3 ) +max { 0 , α− cos ( hxi , hv ) + cos ( hxi , hv ′ ) } ] where α is the margin between the similarities of a positive pair and a negative pair . Different from previous methods [ 7 ; 32 ] that exploit sentence-level contrastive loss , we follow [ 68 ] to construct a token-level contrastive loss ( triplet margin loss ) that grounds the visual information to each contextualized token output . This fine-grained contrastive loss will help the token-level knowledge distillation in Sec . 3.4 . Masked Language Modeling . For better language understanding in our teacher model , we follow BERT [ 19 ] to use masked language modeling ( MLM ) objective ( Fig . 2 ( b ) ) . By replacing 15 % of tokens in x with a special token [ MASK ] , we obtain a masked text xmasked with the same length . The model takes xmasked as input and learns to predict the tokens by minimizing the negative loglikelihoods : LMLM ( x , xmasked ) = − ∑ i∈Mask log p ( xi | xmasked ) , where Mask refers to the indices of masked tokens .

Summary: The paper proposed to distill the large-scale pre-trained video-language multi-modal knowledge into the language modeling. Different knowledge distillation objectives are applied. In the downstream task, the distilled language model shows performance improvement over the conventional language model.