Attention Bottlenecks for Multimodal Fusion
Year: 2021

Abstract:
Humans perceive the world by concurrently processing and fusing highdimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (‘late-fusion’) is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses ‘fusion bottlenecks’ for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense relevant information in each modality and share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

heading: 1 Introduction
text: Simultaneous multimodal sensations are a crucial enabler of human perceptual learning [50]. For artificial learning systems, however, designing a unified model for modality fusion is challenging due to a number of factors: (i) variations in learning dynamics between modalities [56], (ii) different noise topologies, with some modality streams containing more information for the task at hand than others, as well as (iii) specialised input representations. The difference in input representations between audio and vision is particularly stark – many state of the art audio classification methods rely on short term Fourier analysis to produce log-mel spectrograms, often using them as inputs to CNN architectures designed for images [26, 48]. These time-frequency representations have different distributions to images – multiple acoustic objects can have energy at the same frequency, and the translation invariances of CNNs may no longer be a desired property (while an acoustic object can be shifted in time, a shift in frequency could alter the meaning entirely). In contrast, the visual stream in a video is three-dimensional (two spatial and one temporal), and while different spatial regions of the image correspond to different objects, there is the unique challenge of high redundancy across multiple frames. Hence input representations, and consequently neural network architectures and benchmarks tend to vary wildly for different modalities. For simplicity, the dominant paradigm for multimodal fusion therefore often consists of an ad-hoc scheme that involves integrating separate audio and visual networks via their output representations or scores i.e. ‘late-fusion’ [22, 44].
In this work, we present a new transformer based model for audiovisual fusion in video. Despite originally being proposed for NLP tasks, there has been recent interest in transformers [54] as universal perceptual models [29], due to their ability to model dense correlations between tokens, at the same time making few assumptions about their inputs (and because continuous perceptual inputs can be tokenised). By dividing dense continuous signals into patches and rasterising them
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
to 1D tokens, transformers have been shown to perform competitively for image (ViT [16]) and video classification (ViViT [6]), and more recently, audio classification (AST [23]). Because these models are able to elegantly handle variable length sequences, a natural first extension would be to feed in a sequence of both visual and auditory patches to a transformer, with minimal changes to the architecture. This ‘early fusion’ model allows free attention flow between different spatial and temporal regions in the image, as well as across frequency and time in the audio spectrogram. While theoretically appealing, we hypothesise that full pairwise attention at all layers of the model is not necessary because audio and visual inputs contain dense, fine-grained information, much of which is redundant. This is particularly the case for video, as shown by the performance of ‘factorised’ versions of [6]. Such a model would also not scale well to longer videos due to the quadratic complexity of pairwise attention with token sequence length. To mitigate this, we propose two methods to restrict the flow of attention in our model. The first follows from a common paradigm in multimodal learning, which is to restrict cross-modal flow to later layers of the network, allowing early layers to specialise in learning and extracting unimodal patterns. Henceforth this is is referred to as ‘mid fusion’ (Fig. 1, middle left), where the layer at which cross-modal interactions are introduced is called the ‘fusion layer’. The two extreme versions of this are ‘early fusion’ (all layers are cross-modal) and ‘late fusion’ (all are unimodal) which we compare to as a baselines. Our second idea (and main contribution), is to restrict cross-modal attention flow between tokens within a layer. We do this by allowing free attention flow within a modality, but force our model to collate and ‘condense’ information from each modality before sharing it with the other. The core idea is to introduce a small set of latent fusion units that form an ‘attention bottleneck’, through which cross-modal interactions within a layer must pass. We demonstrate that this ‘bottlenecked’ version, which we name Multimodal Bottleneck Transformer (MBT), outperforms or matches its unrestricted counterpart, but with lower computational cost.
Concretely, we make the following contributions: (i) We propose a new architecture (MBT) for audiovisual fusion. Our model restricts the flow of cross-modal information between latent units through tight fusion ‘bottlenecks’, that force the model to collect and ‘condense’ the most relevant inputs in each modality (and therefore share only that which is necessary with the other modality). This avoids the quadratic scaling cost of full pairwise attention, and leads to performance gains with less compute; (ii) We apply MBT to image and spectogram patches (Fig. 2), and explore a number of ablations related to the fusion layer, the sampling of inputs and data size; and finally (iii) We set the new state-of-the-art for video classification across a number of popular audio-visual benchmarks, including AudioSet [21], Epic-Kitchens100 [12] and VGGSound [10]. On the Audioset dataset, we outperform the current state of the art by 5.9 mAP (12.7% relative improvement).

heading: 2 Related work
text: Audiovisual learning: Audiovisual multimodal learning has a rich history, both before and during the deep learning era [47]. Given the limited available data and computational resources, early work focused on relatively simple early-stage (e.g. stacking hand-designed features) and late-stage
(e.g. score fusion) techniques [11]. Deep learning has allowed more sophisticated strategies in which modality-specific or joint latents are implicitly learned to mediate the fusion. The result has enabled major advances in a range of downstream supervised audiovisual tasks [43, 34, 17]. In the supervised setting, multiple modality-specific convolution networks can be jointly trained, whose intermediate activations are then combined by summation [32] or via ‘lateral connections’ [57]. In the unsupervised setting, audiovisual learning is commonly used to learn good unimodal representations, with a popular pretraining task being to synchronise signals from different modalities via a contrastive loss [4, 5, 7, 44, 30, 2, 3], however each modality is usually encoded separately under this setup. Multimodal transformers: The self attention operation of transformers provides a natural mechanism to connect multimodal signals. Multimodal transformers have been applied to various tasks including audio enhancement [17, 53], speech recognition [24], image segmentation [58, 53], crossmodal sequence generation [39, 37, 49], image and video retrieval [25, 20, 8], visual navigation [46] and image/video captioning/classification [41, 52, 51, 36, 28]. For many works, the inputs to transformers are the output representations of single modality CNNs [35, 20] – unlike these works we use transformer blocks throughout, using only a single convolutional layer to rasterise 2D patches. The tokens from different modalities are usually combined directly as inputs to the transformers [38], for example, the recently released Perceiver model [29] introduces an iterative attention mechanism which takes concatenated raw multimodal signals as inputs, which corresponds to our ‘early fusion’ baseline. In contrast, we carefully examine the impact of different modality fusion strategies, including limiting cross-modal attention flow to later layers of our model, and ‘channeling’ cross-modal connections through bottlenecks in our proposed Multimodal Bottleneck Transformer (MBT).

heading: 3 Multimodal fusion transformers
text: In this section we describe our proposed Multimodal Bottleneck Transformer (MBT). We begin by summarising the recently proposed Vision Transformer (ViT) [16] and Audio Spectrogram Transformer (AST) [23], developed for image and audio classification respectively, in Sec. 3.1. We then describe our extension to the audio-visual fusion case. We discuss three different token fusion strategies (Sec. 3.2), and finally discuss the fusion pathway in the entire model (Sec. 3.3), which involves restricting multimodal fusion to certain layers of the model.

heading: 3.1 The ViT and AST architectures
text: Vision Transformer (ViT) [16] (and a recent extension to audio – Audio Spectrogram Transformer (AST) [23]) adapts the Transformer architecture [54], originally designed for natural language processing, to process 2D inputs with minimal changes. The key insight is to extract N nonoverlapping patches from the RGB image (or the audio spectrogram), xi ∈ Rh×w, and convert them into a series of 1D tokens zi ∈ Rd, as follows:
z = g(x;E, zcls) = [zcls,Ex1,Ex2, ...,ExN ] + p. (1)
Here, E is a linear projection mapping each token to Rd, zcls is a special token prepended to this sequence so that its representation at the final layer can be passed to a classifier for classification tasks [15], and p ∈ R(N+1)×d is a learned positional embedding added to the tokens to retain positional information (as all subsequent self-attention operations are permutation invariant).
The tokens are then passed through an encoder consisting of a sequence of L transformer layers. Each transformer layer consists of Multi-Headed Self-Attention (MSA), Layer Normalisation (LN) and Multilayer Perceptron (MLP) blocks applied using residual connections. We denote a transformer layer, zl+1 = Transformer(zl) as
yl = MSA(LN(zl)) + zl (2)
zl+1 = MLP(LN(yl)) + yl. (3) Here, the MSA operation [54] computes dot-product attention [54] where the queries, keys and values are all linear projections of the same tensor, MSA(X) = Attention(WQX,WKX,WV X). We further define Multi-Headed Cross Attention (MCA) between two tensors, X and Y, where X forms the query and Y forms the keys and values which are used to reweight the query as MCA(X,Y) = Attention(WQX,WKY,WV Y). This will be used in our multimodal case, as described next.

heading: 3.2 Multimodal transformer
text: We now describe our extension to the multimodal case. We begin by discussing three different token fusion strategies.

heading: 3.2.1 Fusion via vanilla self-attention
text: We begin by describing a ‘vanilla’ fusion model, which simply consists of the regular transformer applied to multimodal inputs. Our method of tokenising video is straightforward – given a video clip of length t seconds, we uniformly sample F RGB frames and convert the audio waveform into a single spectrogram. We then embed each frame and the spectrogram independently following the encoding proposed in ViT [16], and concatenate all tokens together into a single sequence.
Formally, if we have extracted a total of Nv RGB patches from all F sampled frames, xrgb ∈ RNv×d, and Na spectrogram patches, xspec ∈ RNa×d, our sequence of tokens is z = [zrgb||zspec] where zrgb = g(xrgb;Ergb, zcls-rgb) and zspec = g(xspec;Espec, zcls-spec). (4)
Here, [zrgb||zspec] denotes the concatenation of the tokens for each modality. We use different projections Ergb and Espec for RGB and spectrogram patches respectively, and prepend a separate classification token for each modality.
Our multimodal encoder then applies a series of transformer layers in the same manner as above. Attention is allowed to flow freely through the network, i.e. each RGB token can attend to all other RGB and spectrogram tokens as follows: zl+1 = Transformer(zl; θ) with model parameters θ. Here Transformer refers to a standard transformer layer with vanilla self-attention blocks.

heading: 3.2.2 Fusion with modality-specific parameters
text: We can generalise this model by allowing each modality to have its own dedicated parameters θrgb and θspec, but still exchange information via the attention mechanism. For this purpose, we define a Cross-Transformer layer:
zl+1rgb = Cross-Transformer(z l rgb, z l; θrgb) (5)
zl+1spec = Cross-Transformer(z l spec, z l; θspec),
where the Cross-Transformer employs the generalised cross-attention operation that takes two sets of inputs z1 and z2 that are not necessarily overlapping. This layer follows the original transformer layer with the difference being that Eq. 2 becomes
yl = MCA(LN(zl1),LN(z l 2)) + z l 1. (6)
Finally, note that we have explicitly defined the parameters, θrgb and θspec of the cross-transformer layers in Eq. 5 as they are different for each modality. However, when θrgb and θspec are equal, (θrgb = θspec = θ), the computation defined in Eq. 5 is equivalent to Sec. 3.2.1.

heading: 3.2.3 Fusion via attention bottlenecks
text: In order to tame the quadratic complexity of pairwise attention, we next introduce a small set of B fusion bottleneck tokens zfsn = [z1fsn, z 2 fsn, . . . , z B fsn] to our input sequence (see Fig. 2). The input sequence is now z = [zrgb||zfsn||zspec]. (7)
We then restrict all cross-modal attention flow in our model to be via these bottleneck tokens. More formally for layer l, we compute token representations as follows:
[zl+1i ||ẑ l+1 fsni ] = Transformer([z l i ||zlfsn]; θi) (8)
zl+1fsn = Avgi(ẑ l+1 fsni ) (9)
Here i indexes each modality, in this case RGB and Spec, and zrgb and zspec can only exchange information via the bottleneck zfsn within a transformer layer. We first create modality specific temporary bottleneck fusion tokens ẑfsni , which are updated separately and simultaneously with audio and visual information (Equation 8). The final fusion tokens from each cross-modal update are then averaged in Equation 9. We also experiment with asymmetric updates for the bottleneck tokens (see appendix) and found performance was robust to this choice. We keep the number of bottleneck tokens in the network to be much smaller than the total number of latent units per modality (B ≪ Nv and B ≪ Na). Because all cross-modal attention flow must pass through these units, these tight ‘fusion’ bottlenecks force the model to condense information from each modality and share that which is necessary. As we show in the experiments, this increases or maintains performance for multimodal fusion, at the same time reducing computational complexity. We also note that our formulation is generic to the type and the number of modalities.

heading: 3.3 Where to fuse: early, mid and late
text: The above strategies discuss fusion within a layer, and in most transformer architectures (such as ViT), every layer consists of an identical set of operations. A common paradigm in multimodal learning, however, is to restrict early layers of a network to focus on unimodal processing, and only introduce cross-modal connections at later layers. This is conceptually intuitive if we believe lower layers are involved in processing low level features, while higher layers are focused on learning semantic concepts – low-level visual features such as edges and corners in images might not have a particular sound signature, and therefore might not benefit from early fusion with audio [57].
This can be implemented with our model as follows: We initially perform vanilla self-attention among tokens from a single modality for Lf layers. Thereafter, we concatenate all latent tokens together, zLf = [z
Lf rgb ||z Lf spec] and pass them through the remaining L− Lf layers where the tokens are fused
according to Sec. 3.2. Here, Lf = 0 corresponds to an ‘early-fusion’ model, Lf = L a ‘late-fusion’ model, and 0 < Lf < L a ‘mid-fusion’ one. More formally, this can be denoted as
zl+1rgb = Transformer(z l rgb; θrgb), z l+1 spec = Transformer(z l spec; θspec) if l < Lf
zl = [zlrgb||zlspec], zl+1 = Multimodal-Transformer(zl; θspec, θrgb) otherwise
where Multimodal-Transformer(·) can refer to either of the 3 fusion strategies described in Sec 3.2.

heading: 3.4 Classification
text: For all model variants described above, we pass output representations of the CLS tokens zLcls-rgb and zLcls-spec to the same linear classifier and average the pre-softmax logits.

heading: 4 Experiments
text: We apply MBT to the task of video classification. In this section we first describe the datasets used to train and test multimodal fusion and their respective evaluation protocols (Sec. 4.1), then discuss implementation details (Sec. 4.2). We then ablate the key design choices in our model (Sec. 4.3), before finally comparing our model to the state of the art (Sec. 4.4).

heading: 4.1 Datasets and evaluation protocol
text: We experiment with three video classification datasets – AudioSet [21], Epic-Kitchens-100 [12] and VGGSound [10], described in more detail below. Results on two additional datasets Moments in Time [42] and Kinetics [31] are provided in the appendix. AudioSet [21] consists of almost 2 million 10-second video clips from YouTube, annotated with 527 classes. Like other YouTube datasets, this is a dynamic dataset (we only use the clips still available online). This gives us 20,361 clips for the balanced train set (henceforth referred to as mini-AudioSet or miniAS) and 18,589 clips for the test set. This test set is exactly the same as recent works we compare to, including Perceiver [29]. Instead of using the 2M unbalanced training set, we train on a (slightly more) balanced subset consisting of 500K samples (AS-500K). Details are provided in the appendix. Because each sample has multiple labels, we train with a binary cross-entropy (BCE) loss and report mean average precision (mAP) over all classes, following standard practice. Epic-Kitchens 100 [12] consists of egocentric videos capturing daily kitchen activities. The dataset consists of 90,000 variable length clips spanning 100 hours. We report results for action recognition following standard protocol [12] - each action label is a combination of a verb and noun, and we predict both using a single network with two ‘heads’, both trained with a cross-entropy loss. The top scoring verb and action pair predicted by the network are used, and Top-1 action accuracy is the primary metric. Actions are mainly short-term (average length is 2.6s with minimum length 0.25s). VGGSound [10] contains almost 200K video clips of length 10s, annotated with 309 sound classes consisting of human actions, sound-emitting objects and human-object interactions. Unlike AudioSet, the sound source for each clip is ‘visually present’ in the video. This was ensured during dataset creation through the use of image classifiers. After filtering clips that are no longer available on YouTube, we end up with 172,427 training and 14,448 test clips. We train with a standard crossentropy loss for classification and report Top-1 and Top-5 classification accuracy.

heading: 4.2 Implementation details
text: Our backbone architecture follows that of ViT [16] identically, specifically we use ViT-Base (ViT-B, L = 12, NH = 12, d = 3072)1 initialised from ImageNet-21K [14], however we note that our method is agnostic to transformer backbone. Unless otherwise specialised, we use B = 4 bottleneck tokens for all experiments with bottleneck fusion. Bottleneck tokens are initialized using a Gaussian with mean of 0 and standard deviation of 0.02, similar to the positional embeddings in the public ViT [16] code. We randomly sample clips of t seconds for training. RGB frames for all datasets are extracted at 25 fps. For AudioSet and VGGSound we sample 8 RGB frames over the sampling window of length t with a uniform stride of length (t × 25)/8. We extract 16 × 16 patches from each frame of size 224 × 224, giving us a total of 8 × 14 × 14 = 1568 patches per video. For Epic-Kitchens (because the segments are shorter), we sample 32 frames with stride 1. Audio for all datasets is sampled at 16kHz and converted to mono channel. Similar to [23], we extract log mel spectrograms with a frequency dimension of 128 computed using a 25ms Hamming window with hop length 10ms. This gives us an input of size 128 × 100t for t seconds of audio. Spectrogram patches are extracted with size 16 × 16, giving us 50 × 8 = 400 patches for 8 seconds of audio. For images we apply the standard data augmentations used in [6] (random crop, flip, colour jitter), and for spectrograms we use SpecAugment [45] with a max time mask length of 192 frames and max frequency mask length of 48 bins following AST [23]. We set the base learning rate to 0.5 and train for 50 epochs, using Mixup [59] with α = 0.3 and stochastic depth regularisation [27] with probability p = 0.3. All models (across datasets) are trained with a batch size of 64, synchronous SGD with momentum of 0.9, and a cosine learning rate schedule with warmup of 2.5 epochs on TPU accelerators using the Scenic library [13]. Inference: Following standard practice, we uniformly sample multiple temporal crops from the clip and average per-view logits to obtain the final result. The number of test crops is set to 4.

heading: 4.3 Ablation analysis
text: In this section we investigate the impact of the different architectural choices in MBT. Unless otherwise specified, we use the mini-AudioSet split for training and report results on the AudioSet eval split. More ablations on backbone size and pretraining initalisation can be found in the appendix.
1L is the number of transformer layers, NH is the number of self-attention heads with hidden dimension d.

heading: 4.3.1 Fusion strategies
text: We implement all the three fusion strategies described in Sec. 3.2: (i) Vanilla self-attention – Unrestricted pairwise attention between all latent units within a layer; (ii) Vanilla cross-attention with separate weights: Same as above, but we now have separate weights for each modality. The latent units are updated via pairwise attention with all other latent units from both modalities; and finally (iii) Bottleneck fusion: Here all cross-modal attention must pass through bottleneck fusion latents. Note that these three fusion strategies only describe attention flow between tokens within a layer. For strategies (ii) and (iii), we also conduct experiments showing the impact of restricting cross-modal attention to layers after a fixed fusion layer Lf . We investigate models with different fusion layers, Lf = 0, 2, 4, 6, 8, 10, 12, and present the results in Fig. 3.2 Sharing weights for both modalities: We first investigate the impact of sharing the encoder weights for both modalities (strategy (i) vs (ii)). The results can be found in Fig. 1 in the appendix. When modalities are fused at earlier layers, using separate encoders improves performance. For models with later fusion layers, performance is similar for both models. We hence use separate modality weights for further experiments. Fusion layer: We then investigate the impact of varying the fusion layer Lf , for the latter two strategies: (ii) Vanilla Cross-Attention and (iii) Bottleneck Fusion. We conduct experiments with Lf = 0, 2, 4, 6, 8, 10, 12. We fix the input span t to 4s and the number of bottleneck tokens B to 4. We conduct 3 runs for each experiment and report mean and std deviation. As can be seen from Fig. 3 (left), ‘mid fusion’ outperforms both early (Lf = 0) and late fusion (Lf = 12), with optimal performance obtained by using fusion layer Lf = 10 for vanilla cross-attention and Lf = 8 for bottleneck attention. This suggests that the model benefits from restricting cross-modal connections to later layers, allowing earlier layers to specialise to learning unimodal features, however still benefits from multiple layers of cross-modal information flow. In appendix D , we confirm that mid fusion outperforms late fusion across a number of different datasets. Attention bottlenecks: In Fig. 3, we also examine the effect of bottleneck attention vs vanilla cross-attention for multimodal fusion. We find that for all values of Lf restricting flow to bottlenecks improves or maintains performance, with improvements more prominent at lower values of Lf . At Lf = 10, both perform similarly, note that at this stage we only have 3 fusion layers in the model. Our best performing model uses attention bottlenecks with Lf = 8, and we fix this for all further experiments. We also compare the amount of computation, measured in GFLOPs, for both fusion strategies (Fig. 3, right). Using a small number of bottleneck tokens (in our experiments B = 4) adds negligible extra computation over a late fusion model, with computation remaining largely constant with varying fusion layer Lf . This is in contrast to vanilla cross-fusion, which has a non-negligible computational cost for every layer it is applied to. We note that for early fusion (Lf = 0), bottleneck fusion outperforms vanilla cross-attention by over 2 mAP, with less than half the computational cost. Number of bottleneck tokens B: We experiment with B = 4, 36, 64, 256 and 1024, and find that performance is relatively consistent (all within 0.5 mAP). We hence fix the number of tokens to B = 4 for all experiments. It is interesting that with such a small number of cross-modal connections through only 4 hidden units (B = 4) at each cross-modal layer, we get large performance gains over late fusion (Fig. 3), highlighting the importance of allowing cross-modal information to flow at multiple layers of the model.

heading: 4.3.2 Input sampling and dataset size
text: In this section we investigate the impact of different modality sampling strategies. We also compare to single modality baselines – the visual-only and audio-only baselines consist of a vanilla transformer model applied to only the RGB or spectrogram patches respectively. Sampling window size t: An advantage of our transformer based model is that we can easily input variable length token sequences. We experiment with varying the sampling window t with the following values t = 2, 4, 6 and 8 seconds (note that all videos in AudioSet are 10s), and show results in Fig. 43. At inference, we uniformly sample multiple windows covering the entire video. While the number of spectrogram patches Na changes with t, we keep the number of RGB patches Nv
2Note that Lf = 12 refers to late fusion, where logits are only aggregated after the classifiers, and neither fusion strategy (ii) nor (iii) is applied, but we show results on the same plot for convenience.
3Averaged over 3 runs. Because error bars are small in the plot we also provide them in Table 3 in the appendix.
fixed by changing the stride of frames (to avoid running out of memory). Our results indicate that the performance of both the audio and audio-visual fusion model increases with input span, however the performance of the visual-only model slightly decreases (we hypothesize that this is due to the increased fixed stride, meaning fewer frames are randomly sampled during training). We fix t = 8s in all further experiments. Synchronous vs asynchronous sampling: Given that auditory and visual events may not always be perfected aligned in videos [32], we also investigate asynchronous sampling of different modalities. Here input windows are sampled independently from the entire video clip for each modality. Results are provided in Fig. 2 in the appendix. We find performance to be largely robust to either case, and so for simplicity we use synchronised sampling for all further experiments. Modality MixUp: While applying Mixup regularization [59] to training, we note that there are two different ways to apply it for multimodal inputs – the standard approach is to sample one set of mixup weights from a Beta distribution using the parameter α, and use it to generate all virtual modality-label pairs [59]. We also explore a modified version which we call modality mixup, which samples an independent weight for each modality. Modality mixup imposes stronger augmentation than standard mixup, leading to a slight improvement (42.6 mAP to 43.9 mAP) on AudioSet. Impact of dataset size: We show the impact of varying the number of training samples in Fig. 5, and find a monotonic increase with dataset size (more steeply for audio-only than visual-only).

heading: 4.4 Results
text: Comparison to single modality performance: We compare MBT to visual-only and audio-only baselines on AudioSet (Table 1), Epic-Kitchens (Table 2) and VGGSound (Table 3). Note we use the best parameters obtained via the ablations above, i.e. bottleneck fusion with t = 8, B = 4, Fl = 8 and modality mixup. For all datasets, multimodal fusion outperforms the higher-performing single modality baseline, demonstrating the value of complementary information. The relative importance of modalities for the classification labels varies (audio-only has higher relative performance for AudioSet and lower for Epic-Kitchens, while both audio and visual baselines are equally strong for VGGSound). This is (unsurprisingly) largely a function of the dataset annotation procedure and positions VGGSound as a uniquely suitable dataset for fusion. We also show that audio-visual fusion
provides slight performance gains for traditionally video only datasets such as Kinetics and Moments in Time (details provided in Appendix C ). We also examine per-class performance on the Audioset dataset (Figures 3 and 4 in the Appendix), and find that for the top 60 classes (ranked by overall performance), audio-visual fusion improves performance over audio only or visual only for almost all (57 out of 60) classes, except for ‘bagpiping’, ‘emergency vehicle’ and ‘didgeridoo’ which have strong audio signatures. For classes such as ‘bicycle’ and ‘shuffling cards’ where audio signals are weaker, fusion improves over the audio-only baseline by over 60% in absolute AP. Comparison to state of the art: We compare MBT to previous fusion methods on AudioSet in Table 1. We outperform all previous works on fusion (even though we only train on a quarter of the training set – 500K samples), including the recently introduced Perceiver [29] which uses early fusion followed by multiple self attention layers, and Attn Audio-Visual [18] which uses self-attention fusion on top of individual modality CNNs. We compare to previous video classification methods on Epic-Kitchens in Table 2, and note that our model outperforms all previous works that use vision only, as well as TBN [32] which uses three modalities - RGB, audio and optical flow. Given VGGSound is
a relatively new dataset, we compare to two existing audio-only works4 (Table 3), and set the first audiovisual benchmark (that we are aware of) on this dataset. Visualisation of attention maps Finally, we compute maps of the attention from the output CLS tokens to the RGB image input space using Attention Rollout [1]. Results on test images for both a vanilla fusion model and MBT trained on Audioset-mini (fusion layer Lf = 8) are shown in Figure 6. We show the attention maps summed over all the frames in the video clip. We note that first, the model focuses on semantically salient regions in the video for audio classification, particularly regions where there is motion that creates or modifies sound, i.e. the mouth of humans making sounds, fingertips on a piano, hands and instruments. This is unlike state of the art sound source localisation techniques trained with images [9], which tend to highlight the entire object. We further note that the attention maps for MBT are more localised to these regions, showing that the tight bottlenecks do force the model to focus only on the image patches that are actually relevant for the audio classification task and which benefit from early fusion with audio.

heading: 5 Conclusion
text: We propose a new transformer architecture (MBT) for audiovisual fusion, and explore a number of different fusion strategies using cross-attention between latent tokens. We propose a novel strategy to restrict cross-modal attention via a small set of fusion ‘bottlenecks’, and demonstrate that this improves performance over vanilla cross-attention at lower computational cost, achieving state of the art results on a number of benchmarks. Future work will involve extending MBT to other modalities such as text and optical flow. Limitations: The fusion layer is a hyperparameter and may need to be tuned specifically for different tasks and datasets. We also only explore fully supervised fusion, and future work will tackle extensions to a self-supervised learning framework. Broader impact: Multimodal fusion strategies are important for machine learning, as fusing complementary information from different modalities can increase robustness when applied to real world applications. We also note that transformers are in general compute-heavy, which can have adverse environmental effects. We propose a token fusion method via bottlenecks that helps reduce computational complexity when applying transformers for multimodal fusion. Finally, we observe that training datasets contain biases that may render models trained on them unsuitable for certain applications. It is thus possible that people use classification models (intentionally or not) to make decisions that impact different groups in society differently, and it is important to keep this in mind
4To fairly compare to these works, we obtain the scores on the full VGGSound test set from the authors, and compute accuracy metrics on our slightly smaller test set as described in Sec. 4.1.
when deploying, analysing and building upon these models.
Acknowledgements: We would like to thank Joao Carreira for helpful discussions on the Perceiver [29].

Review summaries:
This paper conducts an experimental study of different fusion methods for multimodal audiovisual transformers, and proposes a new fusion mechanism. They mix ordinary transformer layers with new fusion attention layers. These new layers perform cross-attention, in which one modality is used and the other is used as the key/values. This takes places using special fusion tokens, which have a function similar to the classification tokens in ordinary transformers. They find that (1) mid-fusion can obtain better results than early/late fusion in both their model and "vanilla" fusion, (2) attention bottlenecks improve performance, (3) they outperform the very recent, state-of-the-art Perceiver model on AudioSet classification. They also conduct additional experiments on VGG-Sound and Epic Kitchens, in which they compare with other state-of-art models. The supplement contains several other datasets.

The authors propose a new Multimodal Bottleneck Transformer (MBT) for audio-visual learning. Taking image and spectrogram patches as inputs, MBT uses self and cross-attention operations to model both unimodal and cross-modal information at multiple layers in the network. Notably, it adopts a small set of ‘bottleneck’ latent units to force the model to collate and condense the most relevant information in each modality and only share what is necessary. Extensive ablation studies and experiments are conducted on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens, and VGGSound.  ***Post-Rebuttal***  I've read comments from fellow reviewers and the rebuttal from the authors.  Thank the authors for considering my comments and responding to my questions. The rebuttal successfully addressed my major concerns. Thus, I would like to keep my initial positive rating.   The authors should add the missing details, discussion, and new results in the rebuttal to the paper. 

For efficient multimodal fusion, the authors proposed the attention bottlenecks for audio-visual classification tasks. Compared to previous works, they emphasize their bottleneck architecture "to collate and condense the most relevant information in each modality." They showed that it also has the advantage of a lower computational cost. They evaluate the proposed method using three competitive methods, vanilla self-attention, vanilla cross-attention, and their attention bottlenecks. They noted that the position of modality fusion was a critical factor among early, mid and late. 

The authors propose a new transformer architecture MBT. The MBT uses additional learnable tokens to summarize multimodal information between different modalities and an attention modification to process the constructed multimodal sequences efficiently. They perform ablations on fusion strategies and comparison with related models on audio-visual datasets (such as AudioSet, Epic Kitchen 100, VGGSound). They show improved results on most benchmarks.