Parameter Efficient Multimodal Transformers for Video Representation Learning

1 INTRODUCTION . Learning multimodal representation from unlabeled videos has received considerable attention ( Baltrušaitis et al. , 2018 ) . Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visual co-occurrence ( Owens & Efros , 2018 ; Owens et al. , 2018 ; Arandjelovic & Zisserman , 2018 ; Ephrat et al. , 2018 ; Gao & Grauman , 2019 ; Alwassel et al. , 2019 ) . However , existing approaches learn localized representations from short videos ( hundreds of milliseconds to just under a few seconds ) , capturing only short-term dependencies in data . While this is useful for certain applications , e.g. , source separation ( Ephrat et al. , 2018 ) and atomic action recognition ( Gu et al. , 2018 ) , learning representation that captures long-term dependencies is equally important , e.g. , for activity recognition ( Kay et al. , 2017 ; Carreira et al. , 2019 ; Sigurdsson et al. , 2016 ) . Unfortunately , processing long videos requires large memory resource and capturing long-term dependencies is a long-standing problem ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ; Vaswani et al. , 2017 ) . In language understanding , strong progress has been made in large-scale learning of contextualized language representations using Transformers ( Vaswani et al. , 2017 ; Howard & Ruder , 2018 ; Peters et al. , 2018 ; Radford et al. , 2018 ; 2019 ; Devlin et al. , 2019 ; Liu et al. , 2019 ; Yang et al. , 2019 ) . Riding on the success of Transformers , several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework ( Sun et al. , 2019b ; Lu et al. , 2019 ) . However , these models are typically not end-to-end trained ; they rely on a language-pretrained BERT ( Devlin et al. , 2019 ) , which is fixed throughout , and train only the visual components . While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal to the vision component , this partial learning setup can be undesirable if the text data comes from different distributions ( of topics , dialects , or foreign languages ) or if we want to apply it to different modalities ( e.g. , audio-visual ) . Unfortunately , end-to-end training of such multimodal Transformer architectures is challenging for most existing compute environments due to the excessive memory requirement . In this work , we make three key contributions . First , we propose an end-to-end trainable bidirectional transformer architecture that learns contextualized audio-visual representations of long videos . Our model , shown in Figure 1 , consists of audio/visual CNNs , audio/visual Transformers , and a multimodal Transformer . The CNNs operate on short ( e.g. , one second ) video clips and are intended to capture short-term dynamics within each modality . The Transformer layers operate on long video sequences ( e.g. , 30 seconds ) , capturing long-term dynamics . To enable end-to-end training , we propose a novel parameter reduction technique that shares parts of weight parameters across Transformers and across layers within each Transformer . We show that this results in up to 97 % parameter reduction , enabling end-to-end training of our model , with a minimal performance degradation . To the best of our knowledge , our work is the first to report end-to-end trained multimodal Transformers , and the first to apply Transformers for audio-visual representation learning . The quality of negative samples is crucial in contrastive learning , which is part of our learning objective . As our second contribution , we propose a content-aware negative sampling strategy that favors negatives sufficiently similar to a positive instance . Our approach measures the similarity by reusing the CNN embeddings obtained during model training , and thus do not introduce extra parameters to learn . We show that this improves performance over the standard sampling strategies . Our third contribution is a systematic evaluation of different modality fusion strategies . Existing works on multimodal BERT ( all using vision-and-language data ) typically apply one fusion strategy without thoroughly comparing with alternatives , e.g. , some works perform early fusion ( Sun et al. , 2019b ; Su et al. , 2020 ) while others perform mid-level fusion ( Lu et al. , 2019 ; Tan & Bansal , 2019 ) . As a result , it is unclear how different fusion methods affect the final performance . In this work , we compare three fusion strategies ( early , mid , late ) and show the superiority of mid-level fusion . To demonstrate our approach , we pretrain our model on long ( 30-second ) video clips from Kinetics700 ( Carreira et al. , 2019 ) and finetune it on various video classification tasks . One benefit of the modular design of our architecture is flexibility : once pretrained , we can use any of the subnetworks for downstream tasks depending on the modalities involved ( audio-only , visual-only , audio-visual ) and video lengths ( short and long ) . To show this , we evaluate our model on UCF101 ( Soomro et al. , 2012 ) and ESC-50 ( Gemmeke et al. , 2017 ) for short-term visual/audio classification , and Charades ( Sigurdsson et al. , 2016 ) and Kinetics-Sounds ( Arandjelovic & Zisserman , 2017 ) for long-term audio-visual action recognition . 2 APPROACH . Figure 1 shows an overview of the proposed model architecture . The input to our model is a sequence of visual clips v1 : T and the corresponding sequence of audio streams a1 : T . For example , each sequence is a 30 second-long video divided into 30 non-overlapping clips ( each clip is one second long ) . We divide our model into three parts with different characteristics , which are explained below . Local Feature Embedding . We feed each of T video clips to a visual CNN fV ( vt ) to obtain xv1 : T ∈ RT×D , and each audio stream to an audio CNN fA ( at ) to obtain xa1 : T ∈ RT×D.1 Intuitively , the CNN outputs are temporally local embeddings as they have access to only a short-range temporal window of the entire video sequence . Thus , they are suitable for representing short-range atomic actions ( e.g. , sit down , raise arms ) that constitute long-range events ( e.g. , gym workout ) . We use the SlowFast network ( Feichtenhofer et al. , 2019 ) with a ResNet-50 backbone ( He et al. , 2016 ) as a visual CNN fV , and a ResNet-50 as an audio CNN fA . The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers . Unimodal Contextualized Embedding . The local feature embeddings capture short-term dynamics but lack long-term contextual information . We use Transformers ( Vaswani et al. , 2017 ) to enrich the embeddings with sequence-level context . We start by learning unimodal contextualized representations using the visual Transformer gV and the audio Transformer gA , respectively . The Transformer consists of L layers , each with two sub-layers : a multi-head attention layer and a feed-forward layer . Given an input sequence of embeddings x ∈ RT×D and A attention heads , the j-th head in the attention layer computes the output embedding sequence aj ∈ RT×γ , γ = D/A as aj = softmax ( QjK > j√ γ ) Vj , Qj = xW q j , Kj = xW k j , Vj = xW v j ( 1 ) where W qj , W k j , W v j ∈ RD×γ are weight matrices for computing the ( query , key , value ) triplet given the input x . This operation is repeated for each attention head , and the outputs are combined ( with concatenation followed by one linear layer with weights W b ∈ RD×D ) , producing a ∈ RT×D . Next , the feed-forward layer takes this intermediate output and computes o ∈ RT×D using a twolayer fully-connected network with weights W c ∈ RD×E and W d ∈ RE×D . The output of each sub-layer is computed using a residual function followed by layer normalization ( Ba et al. , 2016 ) , i.e. , LayerNorm ( x+ Sublayer ( x ) ) . In this work , we set the number of layers L = 6 , the number of attention heads A = 12 , the feature dimension D = 768 and the intermediate dimension E = 3072 . For simplicity , we use this design for all layers across all three Transformers in our model . Before feeding local embeddings xv and xa to unimodal Transformers , we augment them with “ positional ” embeddings . Specifically , we append to the beginning of each sequence a special vector BOS ( beginning of sequence ) , i.e. , xv0 for visual and x a 0 for audio streams ; their dimensions are same as xvt and x a t , respectively . We also define positional embeddings p0 : T encoding time indices ( we call this “ time ” embedding ) . This is necessary to preserve information about temporal ordering of local feature embeddings , which is otherwise lost in Eqn . 1 . We combine them via layer normalization , uvt = LayerNorm ( x v t + p v t ) , u a t = LayerNorm ( x a t + p a t ) , ∀t ∈ [ 0 , T ] ( 2 ) We initialize { xv0 , xa0 , pv0 : T , pa0 : T } to the normal distribution and train them with the rest of the model . We feed the augmented visual embeddings into the visual Transformer gV and obtain yv0 : T = gV ( u v 0 : T ) , and similarly obtain y a 0 : T = gA ( u a 0 : T ) . The embeddings at each time step has a direct access to the entire input sequence regardless of their position ( it has a one-step signal path during forward and backward inference ) . Multiple layers of such feature transformation thus allow the resulting embedding to be deeply contextualized in the time dimension . We denote the output embeddings corresponding to the BOS positions by BOSvg = y v 0 and BOS a g = y a 0 , and designate them as the summary embeddings representing the sequence of each modality . Multimodal Contextualized Embedding . The unimodal embeddings capture long-term temporal context but miss out on cross-modal information . The final step in forward inference is to use a multimodal Transformer hAV to obtain embeddings contextualized in the audio-visual space . We first augment the embeddings yv0 : T and y a 0 : T with modality and time embeddings . The modality embeddings mv and ma are vectors of the same dimension as yvt and y a t , respectively . We share m v ( and ma ) across all the unimodal embeddings yv0 : T ( and y a 0 : T ) ; thus , they add modality-discriminative information to the Transformer . We also add time embeddings p0 : T as before ; however , unlike in 1For notational simplicity , we drop the subscripts to refer to the entire sequence unless distinction is necessary . the previous step , we share the same p0 : T between embeddings from the two modalities to correctly indicate the time indices . We augment the modality and time embeddings via layer normalization , wvt = LayerNorm ( y v t + pt + m v ) , wat = LayerNorm ( y a t + pt + m a ) , ∀t ∈ [ 0 , T ] ( 3 ) We feed the augmented visual embeddings wv0 : T and audio embeddings w a 0 : T to the multimodal Transformer hAV , one after another , and obtain z0 : ( 2T+1 ) = hAV ( [ wv0 : T ; w a 0 : T ] ) . We again denote the output embeddings corresponding to the BOS positions by BOSvh = z v 0 ( = z0 ) and BOS a h = z a 0 ( = zT+1 ) , and use them as summary embeddings encoding multimodal context . We emphasize the importance of feeding wv0 : T and w a 0 : T one after another . An alternative would be concatenating them before feeding them to hAV and obtaining an output z0 : T ( instead of z0 : ( 2T+1 ) ) . However , this restricts the Transformer to access audio-visual embeddings only from the same time slices , which could be problematic when there is a temporally asynchronous relationship between the two modalities ( e.g. , a visual clip matches with sound captured a few times steps before ) ( Kazakos et al. , 2019 ; Morgado et al. , 2020 ) . By arranging the two sequences one after the other , the Transformer can mix-and-match appropriate audio-visual embeddings in an asynchronous manner . Another practical concern with the alternative approach is that it significantly increases the model size ; the weight matrices Wq , Wk , Wv grow quadratically with the input feature dimension D. Serializing the input resolves both issues .

Summary: This paper studies modeling and training choices when designing a single model based on ConvNets and transformers for audio-visual representation learning. It proposes ablations for which weights/layers to share across modalities, when/where to fuse/join both modalities, and other modeling details (that matter). It also completes pre-training with 3 InfoNCE (audio-audio, visual-visual, audio-visual) with a binary classification loss about if two pairs of audio-visual are from the same or different videos. As strategies for negative sampling in audio and videos are different, it proposes to sample negatives that are similar in the ConvNets' embeddings. The models are pretrained on Kinetics-700 and AudioSet and evaluated on UCF101, ESC-50, and Kinectics-Sounds.