VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer

1 Introduction . Language learning can be aided by grounded visual cues , as they provide powerful signals for modeling a vastness of experiences in the world that can not be documented by text alone [ 5 ; 29 ; 4 ] . While the recent trend of large-scale language model pretraining indirectly provides some world knowledge from text , most large text corpora ( e.g. , Wikipedia ) do not provide enough multi-modal grounding information . Previous works have explored multiple ways of grounding language to visual information such as constructing a common vector space [ 38 ; 7 ] and supervising the model with token-wise generated vision labels [ 68 ] . However , the widely-used image-text datasets ( e.g. , MS COCO [ 48 ] ) are much smaller than text-only corpora in terms of word counts and vocabulary diversity for language learning . The recent method of ‘ vokenization ’ [ 68 ] is a promising initial step towards addressing this problem by supervising language models with weakly-aligned vision-language groundings . Firstly , an imagetext matching model retrieves a corresponding image to each text token in a sentence . Then a language model learns to predict the selected image ( called ‘ voken ’ ) for each text token . This can be seen as a knowledge distillation ( KD ) process [ 33 ] from a vision-language grounding model to a language 1Code and models : https : //github.com/zinengtang/VidLanKD 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . model . Although the voken classification task helps the language model to improve on natural language understanding ( NLU ) tasks , there exist several limitations : ( 1 ) images can not faithfully convey word meanings that require more activity-based and physical commonsense knowledge . ( 2 ) the voken supervision suffers from approximation/quantization error of the text-to-image retrieval . To address these problems , we propose a novel Video-and-Language Knowledge Distillation method , named VIDLANKD . Our teacher model consists of a video encoder and a language encoder . They are jointly trained with a video-language contrastive learning objective and a masked language modeling ( MLM ) objective on a multi-modal dataset ( see Fig . 1 ) . Then , we transfer the knowledge of the frozen teacher language encoder to a student language model by minimizing the distance between contextualized text representations of two models on a text dataset . For this , we propose to use different KD objectives including neuron selectivity transfer ( NST ) [ 34 ] and contrastive representation distillation ( CRD ) [ 71 ] that avoid the approximation error from voken assignments [ 68 ] . For cross-modal pretraining of our teacher model , we use HowTo100M [ 54 ] , a large-scale video dataset which has more diverse vocabulary and richer world commonsense ( e.g. , physical and temporal ) knowledge compared to MS COCO image dataset . In our experiments , student language models learned with the proposed video-language KD objectives outperform the baseline text-pretrained language models and the models distilled with vokenization , on several diverse natural language understanding benchmarks including GLUE [ 73 ] , SQuAD [ 61 ] , and SWAG [ 79 ] . We also show comprehensive ablation studies on video encoders , student KD objectives , teacher pretraining objectives , and video vs. image-based pretraining . Furthermore , we empirically illustrate that our model successfully learns linguistic world knowledge and physical/temporal commonsense abilities from video , by showing improved performances on the GLUE-diagnostics [ 73 ] , PIQA [ 6 ] , and TRACIE [ 82 ] datasets . Overall , our contributions are : ( 1 ) a novel cross-modal knowledge distillation method for improving natural language understanding , ( 2 ) using rich video-text data which can overcome the limitations of image vokenization , ( 3 ) empirical improvements on several language understanding benchmarks and studying different knowledge distillation methods , and ( 4 ) analysis on linguistic/physical/temporal knowledge learned from videos and ablation studies on the effectiveness of proposed components . 2 Related Work . 2.1 Knowledge Distillation . Knowledge distillation ( KD ) [ 33 ] is the process of transferring knowledge from a teacher model to a student model . It has been successfully used in a wide range of applications , such as machine translation [ 40 ] , visual recognition [ 31 ] , speech recognition [ 10 ] , and recommendation systems [ 69 ] . Recent works advanced the field of knowledge distillation by proposing new architectures [ 77 ; 80 ; 1 ; 55 ] and objectives [ 34 ; 14 ] . While many KD works study the problem of knowledge transfer within the same modality , crossmodal knowledge distillation [ 27 ; 20 ; 71 ] tackles the knowledge transfer across different modalities . Gupta et al . [ 27 ] transfers the knowledge of a model trained with RGB images to another model for depth maps and optical flow . Do et al . [ 20 ] proposes a KD method for visual question answering [ 2 ] , where the trilinear ( image-question-answer ) relational representation of a teacher model is transferred to a bilinear ( image-question ) student model . Tian et al . [ 71 ] combines contrastive learning and knowledge distillation to improve the knowledge transfer between different modalities . Our VIDLANKD transfers the knowledge of a multi-modal teacher model learned from a video dataset to a student language model that tackles natural language understanding tasks . 2.2 Language Pretraining . Large-scale pretraining of contextualized language models has seen huge success in natural language processing in recent years . ELMo [ 57 ] proposes to pretrain and fine-tune a large recurrent language model , which improves performance on a diverse set of downstream natural language processing tasks . BERT [ 19 ] improves the scalability of the pretrain/fine-tune framework by using a transformer [ 72 ] language model with a masked language modeling objective . Since then , pretraining of transformer language models has been extensively explored [ 49 ; 78 ; 44 ; 22 ; 64 ; 59 ; 16 ] for various natural language understanding [ 61 ; 74 ; 79 ; 73 ] and generation tasks [ 24 ; 62 ; 61 ; 63 ] . 2.3 Multi-modal Pretraining . Following the success of language pretraining with transformer models , pretraining of image-text [ 67 ; 51 ; 13 ; 47 ; 83 ; 45 ] and video-text [ 66 ; 54 ; 85 ; 53 ; 46 ; 70 ] multi-modal transformers have achieved improvements on numerous multi-modal downstream tasks [ 2 ; 76 ; 84 ] . The multi-modal transformers take both visual and textual inputs and are pretrained on image-text or video-text pairs with multimodal masked language modeling objectives . Despite the success on multi-modal downstream tasks , Tan and Bansal [ 68 ] finds that the multi-modal pretraining does not improve ( and sometimes even harms ) the language understanding performance . This is because the scale and diversity of text vocabulary of image-text and video-text datasets are usually smaller than those of text datasets . To utilize the rich vocabulary of text dataset , our VIDLANKD transfers the knowledge of pretrained multi-modal model to a student language model with a large text dataset . 2.4 Visually-Grounded Language Learning . A series of works explore using visual information to aid language understanding and generation tasks including co-reference resolution [ 42 ; 15 ] , machine translation [ 81 ] , bilingual lexicon learning [ 39 ] , and multi-modal contrastive learning [ 47 ] . Vokenization [ 68 ] proposes the visually-supervised language model , which is closest to our work . Vokenization proposes to supervise a language model to predict a visualized token , called ‘ voken ’ for each input text token . Vokens are obtained by a contextualized token-to-image matching model , pretrained on a MS COCO image captioning dataset [ 12 ] . In this work , we experiment with alternative objectives which avoid the approximation error from finite voken assignments . In addition , we use HowTo100M [ 54 ] video dataset , which provides a more diverse vocabulary as well as richer world commonsense and physical reasoning knowledge . 3 Video-Language Knowledge Distillation . 3.1 Method Overview . We aim to learn a better language representation with the knowledge distilled from visual information . For this , we leverage two kinds of datasets : the aligned muti-modal dataset , DVL : { ( x , v ) } ( e.g. , HowTo100M [ 54 ] ) ; and the text dataset , DL : { x } ( e.g. , Wikipedia ) , where x is a sentence and v is a video paired with x . Our knowledge transfer is done in two stages : ( 1 ) cross-modal pretraining of a teacher model , MT , on multi-modal data DVL ( Eq . 1 ) ( 2 ) distilling the knowledge of teacher model to a student model , MS , on text data DL ( Eq . 2 ) . We illustrate our two-stage knowledge transfer method in Fig . 1. min θT E x , v∼DVL LT ( MT , x , v ) ( 1 ) min θS E x∼DL LKD ( MT , MS , x ) ( 2 ) Our teacher model MT consists a language model LMT and a visual encoder VT . Both LMT and VT have transformer [ 72 ] architectures , where LMT takes text tokens x and VT takes video frame features v as inputs . Our student model MS is a transformer language model LMS sharing the same architecture with teacher language model LMT . As illustrated in Fig . 1 ( a ) , we first train teacher models LMT and VT with contrastive learning and masked language modeling . Then , we distill the knowledge of teacher models to student model LMS as in Fig . 1 ( b ) . In the following subsections , we discuss the detailed training procedure of teacher ( Sec . 3.2 , Fig . 2 ) and student models ( Sec . 3.3 , Fig . 3 ) . 3.2 Teacher Model . We train our teacher model on a multi-modal dataset with two objectives , i.e. , video-language contrastive learning ( Fig . 2 ( a ) ) and masked language modeling ( Fig . 2 ( b ) ) : LT = LCT + LMLM 2 Architecture . As shown in Figure 2 , our teacher model MT consists of a language encoder LMT and a visual encoder VT . Both LMT and VT have similar transformer architecture.3 For each sentence x , we tokenize it and append a special token [ CLS ] that represents the entire sentence following Devlin et al . [ 19 ] . LMT takes x and outputs contextualized representation hx = { hx [ CLS ] , hx1 · · ·hx|x| } . For each video v , we extract frame-level features ev with an off-the-shelf image encoder ( see more details in Sec . 4.2 ) . Note that the parameters of the image encoder are not updated to save computation . We feed the frame features ev = { ev1 · · · ev|v| } to our visual encoder V T to get contextualized video frame features hv = { hv1 · · ·hv|v| } . We get the final video representation hv by temporally averaging frame-level features : hv = 1|v| ∑|v| i=1 h v i . Different from Tan and Bansal [ 68 ] , both LM T and VT parameters are trained from scratch . Video-Language Contrastive Learning . To learn multi-modal grounding , we use a contrastive learning objective that encourages a closer distance between representations of aligned video-text pairs than unaligned pairs , as shown in Fig . 2 ( a ) . For each x , we randomly sample another text x′ from its batch with x′ ̸= x . Similarly , for each v , we randomly sample another video v′ from its 2In our experiments , different weights over the objectives did not significantly change the results . 3In our experiments , we use the BERT architecture with two different configurations : 12 layers/768 hidden dimensions ( BERT12L/768H = BERTBASE ) and 6 layers/512 hidden dimensions ( BERT6L/512H ) . batch with v′ ̸= v. Then , we use hinge loss max { 0 , α− pos + neg } on cosine similarities : LCT ( x , x′ , v , v′ ) = |x|∑ i [ max { 0 , α− cos ( hxi , hv ) + cos ( hx ′ i , h v ) } ( 3 ) +max { 0 , α− cos ( hxi , hv ) + cos ( hxi , hv ′ ) } ] where α is the margin between the similarities of a positive pair and a negative pair . Different from previous methods [ 7 ; 32 ] that exploit sentence-level contrastive loss , we follow [ 68 ] to construct a token-level contrastive loss ( triplet margin loss ) that grounds the visual information to each contextualized token output . This fine-grained contrastive loss will help the token-level knowledge distillation in Sec . 3.4 . Masked Language Modeling . For better language understanding in our teacher model , we follow BERT [ 19 ] to use masked language modeling ( MLM ) objective ( Fig . 2 ( b ) ) . By replacing 15 % of tokens in x with a special token [ MASK ] , we obtain a masked text xmasked with the same length . The model takes xmasked as input and learns to predict the tokens by minimizing the negative loglikelihoods : LMLM ( x , xmasked ) = − ∑ i∈Mask log p ( xi | xmasked ) , where Mask refers to the indices of masked tokens .

Summary: The paper proposes to improve language understanding using a video-language knowledge distillation method. Compared to the previous state of the art approach, “Vokenization”, they use a large multi-modal video dataset for video-language pre-training to accommodate larger vocabulary needed for better language understanding. For their knowledge distillation based approach, the teacher is the pre-trained model on the multi-modal dataset whose weights are distilled to the student language model trained only on a text dataset. This also avoids the approximation error in the vokenization approach. They perform extensive experiments and improve performance on several downstream language understanding benchmarks (GLUE, SQuAD and SWAG) and also show the commonsense knowledge and reasoning learned from videos.