Attention Bottlenecks for Multimodal Fusion

1 Introduction . Simultaneous multimodal sensations are a crucial enabler of human perceptual learning [ 50 ] . For artificial learning systems , however , designing a unified model for modality fusion is challenging due to a number of factors : ( i ) variations in learning dynamics between modalities [ 56 ] , ( ii ) different noise topologies , with some modality streams containing more information for the task at hand than others , as well as ( iii ) specialised input representations . The difference in input representations between audio and vision is particularly stark – many state of the art audio classification methods rely on short term Fourier analysis to produce log-mel spectrograms , often using them as inputs to CNN architectures designed for images [ 26 , 48 ] . These time-frequency representations have different distributions to images – multiple acoustic objects can have energy at the same frequency , and the translation invariances of CNNs may no longer be a desired property ( while an acoustic object can be shifted in time , a shift in frequency could alter the meaning entirely ) . In contrast , the visual stream in a video is three-dimensional ( two spatial and one temporal ) , and while different spatial regions of the image correspond to different objects , there is the unique challenge of high redundancy across multiple frames . Hence input representations , and consequently neural network architectures and benchmarks tend to vary wildly for different modalities . For simplicity , the dominant paradigm for multimodal fusion therefore often consists of an ad-hoc scheme that involves integrating separate audio and visual networks via their output representations or scores i.e . ‘ late-fusion ’ [ 22 , 44 ] . In this work , we present a new transformer based model for audiovisual fusion in video . Despite originally being proposed for NLP tasks , there has been recent interest in transformers [ 54 ] as universal perceptual models [ 29 ] , due to their ability to model dense correlations between tokens , at the same time making few assumptions about their inputs ( and because continuous perceptual inputs can be tokenised ) . By dividing dense continuous signals into patches and rasterising them 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to 1D tokens , transformers have been shown to perform competitively for image ( ViT [ 16 ] ) and video classification ( ViViT [ 6 ] ) , and more recently , audio classification ( AST [ 23 ] ) . Because these models are able to elegantly handle variable length sequences , a natural first extension would be to feed in a sequence of both visual and auditory patches to a transformer , with minimal changes to the architecture . This ‘ early fusion ’ model allows free attention flow between different spatial and temporal regions in the image , as well as across frequency and time in the audio spectrogram . While theoretically appealing , we hypothesise that full pairwise attention at all layers of the model is not necessary because audio and visual inputs contain dense , fine-grained information , much of which is redundant . This is particularly the case for video , as shown by the performance of ‘ factorised ’ versions of [ 6 ] . Such a model would also not scale well to longer videos due to the quadratic complexity of pairwise attention with token sequence length . To mitigate this , we propose two methods to restrict the flow of attention in our model . The first follows from a common paradigm in multimodal learning , which is to restrict cross-modal flow to later layers of the network , allowing early layers to specialise in learning and extracting unimodal patterns . Henceforth this is is referred to as ‘ mid fusion ’ ( Fig . 1 , middle left ) , where the layer at which cross-modal interactions are introduced is called the ‘ fusion layer ’ . The two extreme versions of this are ‘ early fusion ’ ( all layers are cross-modal ) and ‘ late fusion ’ ( all are unimodal ) which we compare to as a baselines . Our second idea ( and main contribution ) , is to restrict cross-modal attention flow between tokens within a layer . We do this by allowing free attention flow within a modality , but force our model to collate and ‘ condense ’ information from each modality before sharing it with the other . The core idea is to introduce a small set of latent fusion units that form an ‘ attention bottleneck ’ , through which cross-modal interactions within a layer must pass . We demonstrate that this ‘ bottlenecked ’ version , which we name Multimodal Bottleneck Transformer ( MBT ) , outperforms or matches its unrestricted counterpart , but with lower computational cost . Concretely , we make the following contributions : ( i ) We propose a new architecture ( MBT ) for audiovisual fusion . Our model restricts the flow of cross-modal information between latent units through tight fusion ‘ bottlenecks ’ , that force the model to collect and ‘ condense ’ the most relevant inputs in each modality ( and therefore share only that which is necessary with the other modality ) . This avoids the quadratic scaling cost of full pairwise attention , and leads to performance gains with less compute ; ( ii ) We apply MBT to image and spectogram patches ( Fig . 2 ) , and explore a number of ablations related to the fusion layer , the sampling of inputs and data size ; and finally ( iii ) We set the new state-of-the-art for video classification across a number of popular audio-visual benchmarks , including AudioSet [ 21 ] , Epic-Kitchens100 [ 12 ] and VGGSound [ 10 ] . On the Audioset dataset , we outperform the current state of the art by 5.9 mAP ( 12.7 % relative improvement ) . 2 Related work . Audiovisual learning : Audiovisual multimodal learning has a rich history , both before and during the deep learning era [ 47 ] . Given the limited available data and computational resources , early work focused on relatively simple early-stage ( e.g . stacking hand-designed features ) and late-stage ( e.g . score fusion ) techniques [ 11 ] . Deep learning has allowed more sophisticated strategies in which modality-specific or joint latents are implicitly learned to mediate the fusion . The result has enabled major advances in a range of downstream supervised audiovisual tasks [ 43 , 34 , 17 ] . In the supervised setting , multiple modality-specific convolution networks can be jointly trained , whose intermediate activations are then combined by summation [ 32 ] or via ‘ lateral connections ’ [ 57 ] . In the unsupervised setting , audiovisual learning is commonly used to learn good unimodal representations , with a popular pretraining task being to synchronise signals from different modalities via a contrastive loss [ 4 , 5 , 7 , 44 , 30 , 2 , 3 ] , however each modality is usually encoded separately under this setup . Multimodal transformers : The self attention operation of transformers provides a natural mechanism to connect multimodal signals . Multimodal transformers have been applied to various tasks including audio enhancement [ 17 , 53 ] , speech recognition [ 24 ] , image segmentation [ 58 , 53 ] , crossmodal sequence generation [ 39 , 37 , 49 ] , image and video retrieval [ 25 , 20 , 8 ] , visual navigation [ 46 ] and image/video captioning/classification [ 41 , 52 , 51 , 36 , 28 ] . For many works , the inputs to transformers are the output representations of single modality CNNs [ 35 , 20 ] – unlike these works we use transformer blocks throughout , using only a single convolutional layer to rasterise 2D patches . The tokens from different modalities are usually combined directly as inputs to the transformers [ 38 ] , for example , the recently released Perceiver model [ 29 ] introduces an iterative attention mechanism which takes concatenated raw multimodal signals as inputs , which corresponds to our ‘ early fusion ’ baseline . In contrast , we carefully examine the impact of different modality fusion strategies , including limiting cross-modal attention flow to later layers of our model , and ‘ channeling ’ cross-modal connections through bottlenecks in our proposed Multimodal Bottleneck Transformer ( MBT ) . 3 Multimodal fusion transformers . In this section we describe our proposed Multimodal Bottleneck Transformer ( MBT ) . We begin by summarising the recently proposed Vision Transformer ( ViT ) [ 16 ] and Audio Spectrogram Transformer ( AST ) [ 23 ] , developed for image and audio classification respectively , in Sec . 3.1 . We then describe our extension to the audio-visual fusion case . We discuss three different token fusion strategies ( Sec . 3.2 ) , and finally discuss the fusion pathway in the entire model ( Sec . 3.3 ) , which involves restricting multimodal fusion to certain layers of the model . 3.1 The ViT and AST architectures . Vision Transformer ( ViT ) [ 16 ] ( and a recent extension to audio – Audio Spectrogram Transformer ( AST ) [ 23 ] ) adapts the Transformer architecture [ 54 ] , originally designed for natural language processing , to process 2D inputs with minimal changes . The key insight is to extract N nonoverlapping patches from the RGB image ( or the audio spectrogram ) , xi ∈ Rh×w , and convert them into a series of 1D tokens zi ∈ Rd , as follows : z = g ( x ; E , zcls ) = [ zcls , Ex1 , Ex2 , ... , ExN ] + p. ( 1 ) Here , E is a linear projection mapping each token to Rd , zcls is a special token prepended to this sequence so that its representation at the final layer can be passed to a classifier for classification tasks [ 15 ] , and p ∈ R ( N+1 ) ×d is a learned positional embedding added to the tokens to retain positional information ( as all subsequent self-attention operations are permutation invariant ) . The tokens are then passed through an encoder consisting of a sequence of L transformer layers . Each transformer layer consists of Multi-Headed Self-Attention ( MSA ) , Layer Normalisation ( LN ) and Multilayer Perceptron ( MLP ) blocks applied using residual connections . We denote a transformer layer , zl+1 = Transformer ( zl ) as yl = MSA ( LN ( zl ) ) + zl ( 2 ) zl+1 = MLP ( LN ( yl ) ) + yl . ( 3 ) Here , the MSA operation [ 54 ] computes dot-product attention [ 54 ] where the queries , keys and values are all linear projections of the same tensor , MSA ( X ) = Attention ( WQX , WKX , WV X ) . We further define Multi-Headed Cross Attention ( MCA ) between two tensors , X and Y , where X forms the query and Y forms the keys and values which are used to reweight the query as MCA ( X , Y ) = Attention ( WQX , WKY , WV Y ) . This will be used in our multimodal case , as described next . 3.2 Multimodal transformer . We now describe our extension to the multimodal case . We begin by discussing three different token fusion strategies . 3.2.1 Fusion via vanilla self-attention . We begin by describing a ‘ vanilla ’ fusion model , which simply consists of the regular transformer applied to multimodal inputs . Our method of tokenising video is straightforward – given a video clip of length t seconds , we uniformly sample F RGB frames and convert the audio waveform into a single spectrogram . We then embed each frame and the spectrogram independently following the encoding proposed in ViT [ 16 ] , and concatenate all tokens together into a single sequence . Formally , if we have extracted a total of Nv RGB patches from all F sampled frames , xrgb ∈ RNv×d , and Na spectrogram patches , xspec ∈ RNa×d , our sequence of tokens is z = [ zrgb||zspec ] where zrgb = g ( xrgb ; Ergb , zcls-rgb ) and zspec = g ( xspec ; Espec , zcls-spec ) . ( 4 ) Here , [ zrgb||zspec ] denotes the concatenation of the tokens for each modality . We use different projections Ergb and Espec for RGB and spectrogram patches respectively , and prepend a separate classification token for each modality . Our multimodal encoder then applies a series of transformer layers in the same manner as above . Attention is allowed to flow freely through the network , i.e . each RGB token can attend to all other RGB and spectrogram tokens as follows : zl+1 = Transformer ( zl ; θ ) with model parameters θ . Here Transformer refers to a standard transformer layer with vanilla self-attention blocks .

Summary: The authors propose a new Multimodal Bottleneck Transformer (MBT) for audio-visual learning. Taking image and spectrogram patches as inputs, MBT uses self and cross-attention operations to model both unimodal and cross-modal information at multiple layers in the network. Notably, it adopts a small set of ‘bottleneck’ latent units to force the model to collate and condense the most relevant information in each modality and only share what is necessary. Extensive ablation studies and experiments are conducted on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens, and VGGSound.  ***Post-Rebuttal***  I've read comments from fellow reviewers and the rebuttal from the authors.  Thank the authors for considering my comments and responding to my questions. The rebuttal successfully addressed my major concerns. Thus, I would like to keep my initial positive rating.   The authors should add the missing details, discussion, and new results in the rebuttal to the paper.