Attention Bottlenecks for Multimodal Fusion

1 Introduction . Simultaneous multimodal sensations are a crucial enabler of human perceptual learning [ 50 ] . For artificial learning systems , however , designing a unified model for modality fusion is challenging due to a number of factors : ( i ) variations in learning dynamics between modalities [ 56 ] , ( ii ) different noise topologies , with some modality streams containing more information for the task at hand than others , as well as ( iii ) specialised input representations . The difference in input representations between audio and vision is particularly stark – many state of the art audio classification methods rely on short term Fourier analysis to produce log-mel spectrograms , often using them as inputs to CNN architectures designed for images [ 26 , 48 ] . These time-frequency representations have different distributions to images – multiple acoustic objects can have energy at the same frequency , and the translation invariances of CNNs may no longer be a desired property ( while an acoustic object can be shifted in time , a shift in frequency could alter the meaning entirely ) . In contrast , the visual stream in a video is three-dimensional ( two spatial and one temporal ) , and while different spatial regions of the image correspond to different objects , there is the unique challenge of high redundancy across multiple frames . Hence input representations , and consequently neural network architectures and benchmarks tend to vary wildly for different modalities . For simplicity , the dominant paradigm for multimodal fusion therefore often consists of an ad-hoc scheme that involves integrating separate audio and visual networks via their output representations or scores i.e . ‘ late-fusion ’ [ 22 , 44 ] . In this work , we present a new transformer based model for audiovisual fusion in video . Despite originally being proposed for NLP tasks , there has been recent interest in transformers [ 54 ] as universal perceptual models [ 29 ] , due to their ability to model dense correlations between tokens , at the same time making few assumptions about their inputs ( and because continuous perceptual inputs can be tokenised ) . By dividing dense continuous signals into patches and rasterising them 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to 1D tokens , transformers have been shown to perform competitively for image ( ViT [ 16 ] ) and video classification ( ViViT [ 6 ] ) , and more recently , audio classification ( AST [ 23 ] ) . Because these models are able to elegantly handle variable length sequences , a natural first extension would be to feed in a sequence of both visual and auditory patches to a transformer , with minimal changes to the architecture . This ‘ early fusion ’ model allows free attention flow between different spatial and temporal regions in the image , as well as across frequency and time in the audio spectrogram . While theoretically appealing , we hypothesise that full pairwise attention at all layers of the model is not necessary because audio and visual inputs contain dense , fine-grained information , much of which is redundant . This is particularly the case for video , as shown by the performance of ‘ factorised ’ versions of [ 6 ] . Such a model would also not scale well to longer videos due to the quadratic complexity of pairwise attention with token sequence length . To mitigate this , we propose two methods to restrict the flow of attention in our model . The first follows from a common paradigm in multimodal learning , which is to restrict cross-modal flow to later layers of the network , allowing early layers to specialise in learning and extracting unimodal patterns . Henceforth this is is referred to as ‘ mid fusion ’ ( Fig . 1 , middle left ) , where the layer at which cross-modal interactions are introduced is called the ‘ fusion layer ’ . The two extreme versions of this are ‘ early fusion ’ ( all layers are cross-modal ) and ‘ late fusion ’ ( all are unimodal ) which we compare to as a baselines . Our second idea ( and main contribution ) , is to restrict cross-modal attention flow between tokens within a layer . We do this by allowing free attention flow within a modality , but force our model to collate and ‘ condense ’ information from each modality before sharing it with the other . The core idea is to introduce a small set of latent fusion units that form an ‘ attention bottleneck ’ , through which cross-modal interactions within a layer must pass . We demonstrate that this ‘ bottlenecked ’ version , which we name Multimodal Bottleneck Transformer ( MBT ) , outperforms or matches its unrestricted counterpart , but with lower computational cost . Concretely , we make the following contributions : ( i ) We propose a new architecture ( MBT ) for audiovisual fusion . Our model restricts the flow of cross-modal information between latent units through tight fusion ‘ bottlenecks ’ , that force the model to collect and ‘ condense ’ the most relevant inputs in each modality ( and therefore share only that which is necessary with the other modality ) . This avoids the quadratic scaling cost of full pairwise attention , and leads to performance gains with less compute ; ( ii ) We apply MBT to image and spectogram patches ( Fig . 2 ) , and explore a number of ablations related to the fusion layer , the sampling of inputs and data size ; and finally ( iii ) We set the new state-of-the-art for video classification across a number of popular audio-visual benchmarks , including AudioSet [ 21 ] , Epic-Kitchens100 [ 12 ] and VGGSound [ 10 ] . On the Audioset dataset , we outperform the current state of the art by 5.9 mAP ( 12.7 % relative improvement ) . 2 Related work . Audiovisual learning : Audiovisual multimodal learning has a rich history , both before and during the deep learning era [ 47 ] . Given the limited available data and computational resources , early work focused on relatively simple early-stage ( e.g . stacking hand-designed features ) and late-stage ( e.g . score fusion ) techniques [ 11 ] . Deep learning has allowed more sophisticated strategies in which modality-specific or joint latents are implicitly learned to mediate the fusion . The result has enabled major advances in a range of downstream supervised audiovisual tasks [ 43 , 34 , 17 ] . In the supervised setting , multiple modality-specific convolution networks can be jointly trained , whose intermediate activations are then combined by summation [ 32 ] or via ‘ lateral connections ’ [ 57 ] . In the unsupervised setting , audiovisual learning is commonly used to learn good unimodal representations , with a popular pretraining task being to synchronise signals from different modalities via a contrastive loss [ 4 , 5 , 7 , 44 , 30 , 2 , 3 ] , however each modality is usually encoded separately under this setup . Multimodal transformers : The self attention operation of transformers provides a natural mechanism to connect multimodal signals . Multimodal transformers have been applied to various tasks including audio enhancement [ 17 , 53 ] , speech recognition [ 24 ] , image segmentation [ 58 , 53 ] , crossmodal sequence generation [ 39 , 37 , 49 ] , image and video retrieval [ 25 , 20 , 8 ] , visual navigation [ 46 ] and image/video captioning/classification [ 41 , 52 , 51 , 36 , 28 ] . For many works , the inputs to transformers are the output representations of single modality CNNs [ 35 , 20 ] – unlike these works we use transformer blocks throughout , using only a single convolutional layer to rasterise 2D patches . The tokens from different modalities are usually combined directly as inputs to the transformers [ 38 ] , for example , the recently released Perceiver model [ 29 ] introduces an iterative attention mechanism which takes concatenated raw multimodal signals as inputs , which corresponds to our ‘ early fusion ’ baseline . In contrast , we carefully examine the impact of different modality fusion strategies , including limiting cross-modal attention flow to later layers of our model , and ‘ channeling ’ cross-modal connections through bottlenecks in our proposed Multimodal Bottleneck Transformer ( MBT ) . 3 Multimodal fusion transformers . In this section we describe our proposed Multimodal Bottleneck Transformer ( MBT ) . We begin by summarising the recently proposed Vision Transformer ( ViT ) [ 16 ] and Audio Spectrogram Transformer ( AST ) [ 23 ] , developed for image and audio classification respectively , in Sec . 3.1 . We then describe our extension to the audio-visual fusion case . We discuss three different token fusion strategies ( Sec . 3.2 ) , and finally discuss the fusion pathway in the entire model ( Sec . 3.3 ) , which involves restricting multimodal fusion to certain layers of the model . 3.1 The ViT and AST architectures . Vision Transformer ( ViT ) [ 16 ] ( and a recent extension to audio – Audio Spectrogram Transformer ( AST ) [ 23 ] ) adapts the Transformer architecture [ 54 ] , originally designed for natural language processing , to process 2D inputs with minimal changes . The key insight is to extract N nonoverlapping patches from the RGB image ( or the audio spectrogram ) , xi ∈ Rh×w , and convert them into a series of 1D tokens zi ∈ Rd , as follows : z = g ( x ; E , zcls ) = [ zcls , Ex1 , Ex2 , ... , ExN ] + p. ( 1 ) Here , E is a linear projection mapping each token to Rd , zcls is a special token prepended to this sequence so that its representation at the final layer can be passed to a classifier for classification tasks [ 15 ] , and p ∈ R ( N+1 ) ×d is a learned positional embedding added to the tokens to retain positional information ( as all subsequent self-attention operations are permutation invariant ) . The tokens are then passed through an encoder consisting of a sequence of L transformer layers . Each transformer layer consists of Multi-Headed Self-Attention ( MSA ) , Layer Normalisation ( LN ) and Multilayer Perceptron ( MLP ) blocks applied using residual connections . We denote a transformer layer , zl+1 = Transformer ( zl ) as yl = MSA ( LN ( zl ) ) + zl ( 2 ) zl+1 = MLP ( LN ( yl ) ) + yl . ( 3 ) Here , the MSA operation [ 54 ] computes dot-product attention [ 54 ] where the queries , keys and values are all linear projections of the same tensor , MSA ( X ) = Attention ( WQX , WKX , WV X ) . We further define Multi-Headed Cross Attention ( MCA ) between two tensors , X and Y , where X forms the query and Y forms the keys and values which are used to reweight the query as MCA ( X , Y ) = Attention ( WQX , WKY , WV Y ) . This will be used in our multimodal case , as described next . 3.2 Multimodal transformer . We now describe our extension to the multimodal case . We begin by discussing three different token fusion strategies . 3.2.1 Fusion via vanilla self-attention . We begin by describing a ‘ vanilla ’ fusion model , which simply consists of the regular transformer applied to multimodal inputs . Our method of tokenising video is straightforward – given a video clip of length t seconds , we uniformly sample F RGB frames and convert the audio waveform into a single spectrogram . We then embed each frame and the spectrogram independently following the encoding proposed in ViT [ 16 ] , and concatenate all tokens together into a single sequence . Formally , if we have extracted a total of Nv RGB patches from all F sampled frames , xrgb ∈ RNv×d , and Na spectrogram patches , xspec ∈ RNa×d , our sequence of tokens is z = [ zrgb||zspec ] where zrgb = g ( xrgb ; Ergb , zcls-rgb ) and zspec = g ( xspec ; Espec , zcls-spec ) . ( 4 ) Here , [ zrgb||zspec ] denotes the concatenation of the tokens for each modality . We use different projections Ergb and Espec for RGB and spectrogram patches respectively , and prepend a separate classification token for each modality . Our multimodal encoder then applies a series of transformer layers in the same manner as above . Attention is allowed to flow freely through the network , i.e . each RGB token can attend to all other RGB and spectrogram tokens as follows : zl+1 = Transformer ( zl ; θ ) with model parameters θ . Here Transformer refers to a standard transformer layer with vanilla self-attention blocks .

Summary: This paper conducts an experimental study of different fusion methods for multimodal audiovisual transformers, and proposes a new fusion mechanism. They mix ordinary transformer layers with new fusion attention layers. These new layers perform cross-attention, in which one modality is used and the other is used as the key/values. This takes places using special fusion tokens, which have a function similar to the classification tokens in ordinary transformers. They find that (1) mid-fusion can obtain better results than early/late fusion in both their model and "vanilla" fusion, (2) attention bottlenecks improve performance, (3) they outperform the very recent, state-of-the-art Perceiver model on AudioSet classification. They also conduct additional experiments on VGG-Sound and Epic Kitchens, in which they compare with other state-of-art models. The supplement contains several other datasets.