Parameter Efficient Multimodal Transformers for Video Representation Learning

1 INTRODUCTION . Learning multimodal representation from unlabeled videos has received considerable attention ( Baltrušaitis et al. , 2018 ) . Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visual co-occurrence ( Owens & Efros , 2018 ; Owens et al. , 2018 ; Arandjelovic & Zisserman , 2018 ; Ephrat et al. , 2018 ; Gao & Grauman , 2019 ; Alwassel et al. , 2019 ) . However , existing approaches learn localized representations from short videos ( hundreds of milliseconds to just under a few seconds ) , capturing only short-term dependencies in data . While this is useful for certain applications , e.g. , source separation ( Ephrat et al. , 2018 ) and atomic action recognition ( Gu et al. , 2018 ) , learning representation that captures long-term dependencies is equally important , e.g. , for activity recognition ( Kay et al. , 2017 ; Carreira et al. , 2019 ; Sigurdsson et al. , 2016 ) . Unfortunately , processing long videos requires large memory resource and capturing long-term dependencies is a long-standing problem ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ; Vaswani et al. , 2017 ) . In language understanding , strong progress has been made in large-scale learning of contextualized language representations using Transformers ( Vaswani et al. , 2017 ; Howard & Ruder , 2018 ; Peters et al. , 2018 ; Radford et al. , 2018 ; 2019 ; Devlin et al. , 2019 ; Liu et al. , 2019 ; Yang et al. , 2019 ) . Riding on the success of Transformers , several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework ( Sun et al. , 2019b ; Lu et al. , 2019 ) . However , these models are typically not end-to-end trained ; they rely on a language-pretrained BERT ( Devlin et al. , 2019 ) , which is fixed throughout , and train only the visual components . While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal to the vision component , this partial learning setup can be undesirable if the text data comes from different distributions ( of topics , dialects , or foreign languages ) or if we want to apply it to different modalities ( e.g. , audio-visual ) . Unfortunately , end-to-end training of such multimodal Transformer architectures is challenging for most existing compute environments due to the excessive memory requirement . In this work , we make three key contributions . First , we propose an end-to-end trainable bidirectional transformer architecture that learns contextualized audio-visual representations of long videos . Our model , shown in Figure 1 , consists of audio/visual CNNs , audio/visual Transformers , and a multimodal Transformer . The CNNs operate on short ( e.g. , one second ) video clips and are intended to capture short-term dynamics within each modality . The Transformer layers operate on long video sequences ( e.g. , 30 seconds ) , capturing long-term dynamics . To enable end-to-end training , we propose a novel parameter reduction technique that shares parts of weight parameters across Transformers and across layers within each Transformer . We show that this results in up to 97 % parameter reduction , enabling end-to-end training of our model , with a minimal performance degradation . To the best of our knowledge , our work is the first to report end-to-end trained multimodal Transformers , and the first to apply Transformers for audio-visual representation learning . The quality of negative samples is crucial in contrastive learning , which is part of our learning objective . As our second contribution , we propose a content-aware negative sampling strategy that favors negatives sufficiently similar to a positive instance . Our approach measures the similarity by reusing the CNN embeddings obtained during model training , and thus do not introduce extra parameters to learn . We show that this improves performance over the standard sampling strategies . Our third contribution is a systematic evaluation of different modality fusion strategies . Existing works on multimodal BERT ( all using vision-and-language data ) typically apply one fusion strategy without thoroughly comparing with alternatives , e.g. , some works perform early fusion ( Sun et al. , 2019b ; Su et al. , 2020 ) while others perform mid-level fusion ( Lu et al. , 2019 ; Tan & Bansal , 2019 ) . As a result , it is unclear how different fusion methods affect the final performance . In this work , we compare three fusion strategies ( early , mid , late ) and show the superiority of mid-level fusion . To demonstrate our approach , we pretrain our model on long ( 30-second ) video clips from Kinetics700 ( Carreira et al. , 2019 ) and finetune it on various video classification tasks . One benefit of the modular design of our architecture is flexibility : once pretrained , we can use any of the subnetworks for downstream tasks depending on the modalities involved ( audio-only , visual-only , audio-visual ) and video lengths ( short and long ) . To show this , we evaluate our model on UCF101 ( Soomro et al. , 2012 ) and ESC-50 ( Gemmeke et al. , 2017 ) for short-term visual/audio classification , and Charades ( Sigurdsson et al. , 2016 ) and Kinetics-Sounds ( Arandjelovic & Zisserman , 2017 ) for long-term audio-visual action recognition . 2 APPROACH . Figure 1 shows an overview of the proposed model architecture . The input to our model is a sequence of visual clips v1 : T and the corresponding sequence of audio streams a1 : T . For example , each sequence is a 30 second-long video divided into 30 non-overlapping clips ( each clip is one second long ) . We divide our model into three parts with different characteristics , which are explained below . Local Feature Embedding . We feed each of T video clips to a visual CNN fV ( vt ) to obtain xv1 : T ∈ RT×D , and each audio stream to an audio CNN fA ( at ) to obtain xa1 : T ∈ RT×D.1 Intuitively , the CNN outputs are temporally local embeddings as they have access to only a short-range temporal window of the entire video sequence . Thus , they are suitable for representing short-range atomic actions ( e.g. , sit down , raise arms ) that constitute long-range events ( e.g. , gym workout ) . We use the SlowFast network ( Feichtenhofer et al. , 2019 ) with a ResNet-50 backbone ( He et al. , 2016 ) as a visual CNN fV , and a ResNet-50 as an audio CNN fA . The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers . Unimodal Contextualized Embedding . The local feature embeddings capture short-term dynamics but lack long-term contextual information . We use Transformers ( Vaswani et al. , 2017 ) to enrich the embeddings with sequence-level context . We start by learning unimodal contextualized representations using the visual Transformer gV and the audio Transformer gA , respectively . The Transformer consists of L layers , each with two sub-layers : a multi-head attention layer and a feed-forward layer . Given an input sequence of embeddings x ∈ RT×D and A attention heads , the j-th head in the attention layer computes the output embedding sequence aj ∈ RT×γ , γ = D/A as aj = softmax ( QjK > j√ γ ) Vj , Qj = xW q j , Kj = xW k j , Vj = xW v j ( 1 ) where W qj , W k j , W v j ∈ RD×γ are weight matrices for computing the ( query , key , value ) triplet given the input x . This operation is repeated for each attention head , and the outputs are combined ( with concatenation followed by one linear layer with weights W b ∈ RD×D ) , producing a ∈ RT×D . Next , the feed-forward layer takes this intermediate output and computes o ∈ RT×D using a twolayer fully-connected network with weights W c ∈ RD×E and W d ∈ RE×D . The output of each sub-layer is computed using a residual function followed by layer normalization ( Ba et al. , 2016 ) , i.e. , LayerNorm ( x+ Sublayer ( x ) ) . In this work , we set the number of layers L = 6 , the number of attention heads A = 12 , the feature dimension D = 768 and the intermediate dimension E = 3072 . For simplicity , we use this design for all layers across all three Transformers in our model . Before feeding local embeddings xv and xa to unimodal Transformers , we augment them with “ positional ” embeddings . Specifically , we append to the beginning of each sequence a special vector BOS ( beginning of sequence ) , i.e. , xv0 for visual and x a 0 for audio streams ; their dimensions are same as xvt and x a t , respectively . We also define positional embeddings p0 : T encoding time indices ( we call this “ time ” embedding ) . This is necessary to preserve information about temporal ordering of local feature embeddings , which is otherwise lost in Eqn . 1 . We combine them via layer normalization , uvt = LayerNorm ( x v t + p v t ) , u a t = LayerNorm ( x a t + p a t ) , ∀t ∈ [ 0 , T ] ( 2 ) We initialize { xv0 , xa0 , pv0 : T , pa0 : T } to the normal distribution and train them with the rest of the model . We feed the augmented visual embeddings into the visual Transformer gV and obtain yv0 : T = gV ( u v 0 : T ) , and similarly obtain y a 0 : T = gA ( u a 0 : T ) . The embeddings at each time step has a direct access to the entire input sequence regardless of their position ( it has a one-step signal path during forward and backward inference ) . Multiple layers of such feature transformation thus allow the resulting embedding to be deeply contextualized in the time dimension . We denote the output embeddings corresponding to the BOS positions by BOSvg = y v 0 and BOS a g = y a 0 , and designate them as the summary embeddings representing the sequence of each modality . Multimodal Contextualized Embedding . The unimodal embeddings capture long-term temporal context but miss out on cross-modal information . The final step in forward inference is to use a multimodal Transformer hAV to obtain embeddings contextualized in the audio-visual space . We first augment the embeddings yv0 : T and y a 0 : T with modality and time embeddings . The modality embeddings mv and ma are vectors of the same dimension as yvt and y a t , respectively . We share m v ( and ma ) across all the unimodal embeddings yv0 : T ( and y a 0 : T ) ; thus , they add modality-discriminative information to the Transformer . We also add time embeddings p0 : T as before ; however , unlike in 1For notational simplicity , we drop the subscripts to refer to the entire sequence unless distinction is necessary . the previous step , we share the same p0 : T between embeddings from the two modalities to correctly indicate the time indices . We augment the modality and time embeddings via layer normalization , wvt = LayerNorm ( y v t + pt + m v ) , wat = LayerNorm ( y a t + pt + m a ) , ∀t ∈ [ 0 , T ] ( 3 ) We feed the augmented visual embeddings wv0 : T and audio embeddings w a 0 : T to the multimodal Transformer hAV , one after another , and obtain z0 : ( 2T+1 ) = hAV ( [ wv0 : T ; w a 0 : T ] ) . We again denote the output embeddings corresponding to the BOS positions by BOSvh = z v 0 ( = z0 ) and BOS a h = z a 0 ( = zT+1 ) , and use them as summary embeddings encoding multimodal context . We emphasize the importance of feeding wv0 : T and w a 0 : T one after another . An alternative would be concatenating them before feeding them to hAV and obtaining an output z0 : T ( instead of z0 : ( 2T+1 ) ) . However , this restricts the Transformer to access audio-visual embeddings only from the same time slices , which could be problematic when there is a temporally asynchronous relationship between the two modalities ( e.g. , a visual clip matches with sound captured a few times steps before ) ( Kazakos et al. , 2019 ; Morgado et al. , 2020 ) . By arranging the two sequences one after the other , the Transformer can mix-and-match appropriate audio-visual embeddings in an asynchronous manner . Another practical concern with the alternative approach is that it significantly increases the model size ; the weight matrices Wq , Wk , Wv grow quadratically with the input feature dimension D. Serializing the input resolves both issues .

Summary: The paper is a nice read. It builds on a line of research on multi-modal video understanding that utilises transformers where these works: 1) fix one of the transformer models (e.g. BERT) and 2) utilise tokens and thus do not train the approach in an end-to-end fashion. This typical trend is due to the memory requirements for training a multi-modal transformer end-to-end.