Attention Bottlenecks for Multimodal Fusion
Authors: Arsha Nagrani
Shan Yang
Anurag Arnab
Aren Jansen
Cordelia Schmid
Chen Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: multimodal
fusion
attention
audiovisual
transformers
video

Abstract: Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio.  Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks.
A common approach for building multimodal models is to simply combine multiple of these modality-specific architectures using late-stage fusion of final representations or predictions ('late-fusion').
Instead, we introduce a novel transformer based architecture that uses 'attention bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention,  these bottlenecks force information between different modalities to pass through a small number of '`bottleneck' latent units, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

Attention Bottlenecks for Multimodal Fusion
Arsha Nagrani
Shan Yang Anurag Arnab Cordelia Schmid Chen Sun
Aren Jansen
(anagrani, shanyang, aarnab, arenjansen, cordelias, chensun)@google.com
Google Research
Abstract
Humans perceive the world by concurrently processing and fusing high- dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-peecifi and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality ('late-fusion' multimodal video classification. Instead, we introduce a novel transformer based architecture that uses 'fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information be- tween different modalities pass through small number of bottleneck latents, requiring the model to collate and condense relevant information in each modality and share what necessary. We find that such strategy improves fusion per- formance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve statc-of-the-ar results on multiple audio-visual classi- fication benchmarks including Audioset, Epic-Kite Kitchens and VGGSound. All code and models will be released.
still dominant paradigm for
Introduction
Simultaneous multimodal sensations are a crucial enabler of human perceptual learning [50]. For artificial learning systems, however, designing unified model for modality fusion is challenging due to a number of factors: (i) variations in learning dynamics between modalities [56], (ii) different noise topologies, with some modality streams containing more information for the task hand than others, as well as (iii) specialised input representations. The difference in input representations between audio and vision is particularly stark many state of the audio classification methods rely on short term Fourier analysis to produce log-mel spectrograms, often using them as inputs to CNN architectures designed for images [26, 48]. These tmme-frquency representations have different distributions to images multiple acoustic objects can have energy at the same frequency, and the translation invariances of CNNs may no longer be a desired property (while an acoustic object can be shifted in time, a shift in frequency could alter the meaning entirely). In contrast, the visual stream in a video is dinenssiia (two spatial and one temporal), and while different spatial regions of the image correspond to different objects, there is the unique challenge of high redundancy across multiple frames. Hence input representations, and consequently neural network architectures and benchmarks tend to vary wildly for different modalities. For simplicity, the dominant paradigm for multimodal fusion therefore often consists an ad-hoc scheme that involves integrating separate audio and visual networks via their output representations or scores i.c. 'late-fusion' [22, 44].
In this work, we present a new transformer based model for audiovisual fusion in video. Despite originally being proposed for NLP tasks, there has been recent interest in transformers [54] as universal perceptual models [29], due to their ability model dense correlations between tokens, at the same time making few assumptions about their inputs (and because continuous perceptual inputs can be tokenised). By dividing dense continuous signals into patches and rasterising them
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Type of token:
Audio
Video
Bottleneck
Layer
Late Fusion
Mid Fusion
Bottleneck Fusion
Bottleneck Mid Fusion
Figure 1: Cross-modal Fusion. Unlike late fusion (left), where no cross-modal information is exchanged in the model until after the classifier, we investigate two pathways for the exchange of cross-modal information. The first is via standard pairwise self attention across all hidden units in layer, but applied only to later layers in the model mid fusion (middle, left). We also propose the use of "fusion bottlenecks' (middle, right) that restrict attention flow within a layer through tight be applied in conjunction (Bottleneck Mid Fusion) for
latent units. Both forms of restriction optimal performance (right). We show B = 2 bottleneck units and hidden units per modality. Grey boxes indicate tokens that receive attention flow from both audio and video tokens.
to ID tokens, transformers have been shown to perform competitively for image (ViT [16]) and video classification (ViViT [6]), and more recently, audio classification (AST [23]). Because these models are able to elegantly handle variable length sequences, a natural first extension would be to feed in sequence of both visual and auditory patches to transformer, with minimal changes to the architecture. This 'early fusion' model allows free attention flow between different spatial and temporal regions in the image, well as across frequency and time in the audio spectrogram. While theoretically appealing, we hypothesise that full pairwise attention at all layers of the model is not necessary because audio and visual inputs contain dense, fine-grained information, much of which is redundant. This is particularly the case for video, as shown by the performance of "factorised" versions of [6]. Such model would also not scale well to longer videos due the quadratic complexity of pairwise attention with token sequence length. To mitigate this, we propose two methods to restrict the flow of attention in our model. The first follows from a common paradigm in multimodal learning. which is to restrict cross- cross-modal flow to later layers of the network, allowing early layers to specialise in learning and extracting unimodal patterns. Henceforth this is is referred to as 'mid fusion' (Fig. 1, middle left), where the layer at which cross-modal interactions are introduced is called the 'fusion layer'. The two extreme versions of this are 'early fusion' (all layers are cross-modal) and 'late fusion' (all are unimodal) which we compare to baselines. Our second idea (and main contribution), is to restrict cross-modal attention flow between tokens within a layer. We do this by allowing free attention flow within a modality. but force our model to collate and "condense" information from each modality before sharing it with the other. The core idea is to introduce a small set of latent fusion units that form "attention bottleneck', through which cross-modal interactions within a layer must pass. We demonstrate that this 'bottlenecked' version, which we name Multimodal Bottleneck Transformer (MBT), outperforms matches its unrestricted counterpart, but with lower computational cost.
Concretely, we make the following contributions: (i) We propose a new architecture (MBT) for audiovisual fusion. Our model restricts the flow of cross-modal information between latent units through tight fusion 'bottlenecks', that force the model to collect and 'condense' the most relevant inputs in each modality (and therefore share only that which is necessary with the other modality). This avoids the quadratic scaling cost full pairwise attention, and leads to performance gains with less compute; (ii) We apply MBT to image and spectogram patches (Fig. 2), and explore a number of ablations related to the fusion layer, the sampling of inputs and data size; and finally (iii) We set the new oftte-of-the-ate for video classification across number of popular audio-visur benchmarks, including AudioSet [21], Sopic-Kitchens 100 [12] and VGGSound [10]. On the Audioset dataset, we outperform the current state of the art by 5.9 mAP (12.7% relative improvement).
2 Related work
Audiovisual learning: Audiovisual multimodal learning has a rich history, both before and during the deep learning era [47]. Given the limited available data and computational resources, early work focused on relatively simple carly-stage (c.g. stacking hand hand-designed features) and late-stage
2

---

avg logits
classifier
classifier
Multimodal Bottleneck Transformer
Multimodal Video
CLS
FSN:
FSNg
CLS
Video Projection E Lut...
Multimodal Bottlenecks
Audio Projection
RGB frame patches
Audio spectrogram patches
Figure 2: A Multimodal Fusion Transformer applied to audiovisual inputs. The input sequence consists of image and spectrogram patches. These are then projected into tokens and appended to special CLS (classification) and FSN (fusion bottleneck) tokens. Our transformer encoder then uses self attention model unimodal information, and restricts cross-modal information flow via cross attention with the bottleneck tokens at multiple layers of the network.
(e.g. score fusion) techniques [11]. Deep learning has allowed more sophisticated strategies in which modality-specinf or joint latents are implicitly learned to mediate the fusion. The result has enabled major advances in range downstream supervised audiovisual tasks [43, 34, 17]. In the supervised setting, multiple modality-specifft convolution networks intermediate activations are then combined by summation [32] or via 'lateral connections' [57]. the unsupervised setting, audiovisual learning is commonly used to learn good unimodal representations, with popular pretraining task being to synchronise signals from different modalities via a contrastive loss [4, 7, 44, 30, 2, however each modality is usually encoded separately under this setup. Multimodal transformers: The self attention operation of transformers provides a natural mecha- nism to connect multimodal signals. Multimodal transformers have been applied to various tasks including audio enhancement [17, 53], speech recognition [24], image segmentation [58, 53], cross- modal sequence generation [39, 37, 49], image and video retrieval [25, 20, 8J, visual navigation [46] and image/video captioninglllassffccttinn [41, 52, 51, 36, 28]. For many works, the inputs to trans- formers are output representations of single modality CNNs [35, 20] unlike these works we use transformer blocks throughout, using only single convolutional layer to rasterise 2D patches. The tokens from different modalities are usually combined directly as inputs to the transformers [38], for example, the recently released Perceiver model [29] introduces an iterative attention mechanism which takes concatenated raw multimodal signals as inputs, which corresponds to our 'early fusion' baseline. In contrast, we carefully examine the impact of different modality fusion strategies, includ- ing limiting cross- modal attention flow to later layers of our model, and 'channeling' cross-modal connections through bottlenecks in proposed Multimodal Bottleneck Transformer (MBT).
be jointly trained, whose
3 Multimodal fusion transformers
In this section we describe our proposed Multimodal Bottleneck Transformer (MBT). We begin by summarising the recently proposed Vision Transformer (ViT) [16] and Audio Spectrogram Transformer (AST) [23], developed for image and audio classification respectively, in Sec. 3.1. We then describe our extension to the audio-visual fusion case. We discuss three different token fusion strategies (Sec. 3.2), and finally discuss the fusion pathway in the entire model (Sec. 3.3), which involves restricting multimodal fusion to certain layers of the model.
3.1 The VIT and AST architectures
Vision Transformer (VIT) [16] (and recent extension to audio Audio Spectrogram Transformer (AST) [23]) adapts the Transformer architecture [54], originally designed for natural language processing, to process 2D inputs with minimal changes. The key insight is to extract N non- overlapping patches from the RGB image (or the audio spectrogram), I; E R fex and convert them into a series of ID tokens 2; Rd follows:
g (x; E, Zdss [zots EaT, Ex ... ExN] + p.
(I)
3

---

Here, E is linear projection mapping each token to R Eets is special token prepended to this sequence so that representation at the final layer can be passed classifier for classification tasks [15], and p E [[N+1)xd is learned positional embedding added to the tokens retain positional information (as all subsequent self-attention operations are permutation invariant).
The tokens then passed through encoder consisting of sequence of L transformer layers. Each transformer layer consists Multi- Headed Self- AttenAtttniion (MSA), Layer Normalisation (LN) and Multilayer Perceptron (MLP) blocks applied using residual connections. We denote transformer layer, z +1 Transformer(z') as
y' = MSA(LN(Z))) + z'
(2)
++1 MLP(LN(y')) + yy
(3)
Here, the MSA operation [54] computes dot producct attention [54] where the queries, keys and values are all linear projections of the same tensor, MSA(X) Attention
k X, W X). We further define Multi-Headed Cross Attention (MCA) between two tensors, X and Y, where X forms the query and Y forms the keys and values which are used to reweight the query as MCA(X, Y) = Attentionnt WKY, WVY). This will be used in our multimodal case, as described next.
3.2 Multimodal transformer
We now describe our extension to the multimodal case. We begin by discussing three different token fusion strategies.
3.2.1 Fusion via vanilla self-attention
We begin by describing a 'vanilla' fusion model, which simply consists of the regular transformer applied to multimodal inputs. Our method of tokenising video is straightforward given video clip of length t seconds, we uniformly sample F RGB frames and convert the audio waveform into a single spectrogram. We then embed each frame and the spectrogram independently following the encoding proposed in ViT [16], and concatenate all tokens together into a single sequence.
Formally, if we have extracted total of N, RGB patches from all F sampled frames, Xagh E R Nox, and N₂ spectrogram patches, X.pec E R No xd our sequence of tokens is
z == [Zigb [Zspce] where Zegh = g X Engh Rets-agh) and Zupec = (xxpeci Espec Rets-spee). (4)
Here, [Zggh] Zspee] denotes the concatenation of the tokens for each modality. We use different projections Eigh and Espee for RGB and spectrogram patches respectively, and prepend a separate classification token for each modality.
Our multimodal encoder then applies a series of transformer layers in the same manner as above. Attention allowed flow freely through the network, i.e. each RGB token attend to all other RGB and spectrogram tokens as follows: 2+1 = Transformer(z' 2 () with model parameters 0. Here Transformer refers standard transformer layer with vanilla self-attention blocks.
3.2.2 Fusion with cedallity-specifie parameters
We can generalise this model by allowing each modality to have its own dedicated parameters 0igh and Ospec. but still exchange information via the attention mechanism. For this purpose, we define Cross-Transfore layer:
Cross- Tross-Trausformer Lngo z'; Orgh)
(5)
rgb
(z/pec+ z': Sspec),
spec
where the Cross-Trass Transformer employs the generalised cross-attention operation that takes two sets of inputs 21 and z2 that are not necessarily overlapping. This layer follows the original transformer layer with the difference being that Eq. 2 becomes
(CA ( + +2/.
Finally, note that we have explicitly defined the parameters, Ozz and Ospec of the cross-transformer layers Eq. as they are different for each modality. However, when 0rgh and Oseec are equal, (gig 0 pppe = 0), the computation defined in Eq. 5 is equivalent to Sec. 3.2.1.

---

3.2.3
Fusion via attention bottlenecks
In order to tame the quadratic complexity of pairwise attention, we next introduce small set of B fusion bottleneck tokens Zisa sequence is now
ba 2 2 to our input sequence (see Fig. 2). The input
Rugh
Z [ppe
We then restrict all cross-modal attention flow in our model to be via these bottleneck tokens. More formally for layer I, we compute token representations as follows:
[2
+1
2 ++1
= Transformer(22111 ( 0;)
(8)
Isn,
Fsn
[+1
(2(
(9)
Here indexes each modality, in this case RGB and Spec, and Zrgh and Zeper can only exchange information via the bottleneck Zism within a transformer layer. We first create modality specific temporary bottleneck fusion tokens Ra,, which are updated separately and simultaneously with audio and visual information (Equation 8). The final fusion tokens from each cross- cross-modal modal update are then averaged in Equation 9. We also experiment with asymmetric updates for the bottleneck tokens (see appendix) and found performance was robust to this choice. We keep the number of bottleneck tokens in the network to be much smaller than the total number of latent units per modality (B << N, and B < Na). Because all cross-mes modall attention flow must pass through these units, these tight 'fusion' bottlenecks force the model to condense information from each modality and share that which is necessary. As we show in the experiments, this increases or maintains performance for multimodal fusion, the same time reducing computational complexity. We also note that our formulation is generic to the type and the number of modalities.
3.3
Where to fuse: early, mid and late
The above strategies discuss fusion within layer, and in most transformer architectures (such as ViT), every layer consists of an identical set of operations. learning, however, is to restrict early layers of a network to focus unimodal processing, and only introduce cross-modal connections at later layers. This is conceptually intuitive if we believe lower layers are involved in processing low level features, while higher layers are focused on learning semantic concepts - low-level visual features such as edges and corners in images might not have a particular sound signature, and therefore might not benefit from early fusion with audio [57].
common paradigm in multimodal
This can be implemented with our model as follows: We initially perform vanilla self-attention among tokens from single modality for Ly layers. Thereafter, concatenate all latent tokens together, 2 1 [2p6pa] and pass them through the remaining L - Ly layers where the tokens are fused according to Sec. 3.2. Here, L = 0 corresponds an 'early-fusion' model, L, = L 'late-fusion' model, and 0 Ls < L 'mid-fusion' one. More formally, this can be denoted
2 2-1 Transformer((e (2/g0: Ogo)s <spe Transformer Oppc))
ifl < Li
rgb
z' = |2/ghl pppoel:
z I Gallimoodal-Traanfoommeee Ospec; Orgh)) otherwise
where
Transformer(()) can refer to either of the 3 fusion strategies described in Sec 3.2.
3.4 Classification
For all model variants described above, we pass output representations of the CLSS tokens 2 zelb and 2d spec to the same linear classifier and average the pre-softmax logits.
L
4 Experiments
We apply MBT to the task of video classification. In this section we first describe the datasets used to train and test multimodal fusion and their respective evaluation protocols (Sec. 4.1), then discuss implementation details (Sec. 4.2). We then ablate the key design choices our model (Sec. 4.3), before finally comparing model to the state of the art (Sec. 4.4).
5

---

4.1 Datasets and evaluation protocol
We experiment with three video classification datasets AudioSet [21], Sppic-Kicchens-le 100 [12] and VGGSound [10], described in more detail below. Results on two additional datasets Moments in Time [42] and Kinetics [31] are provided in the appendix.
AudioSet [21] consists of almost 2 million 10-second video clips from YouTube, annotated with 527 classes Like other You' Tube datasets, this is a dynamic dataset (we only use the clips still available online). This gives us 20,361 clips for the balanced train set (henceforth referred to as mini-AudioSet or miniAS) and 18,589 clips for the test set. This test set is exactly the same recent works we compare to, including Perceiver [29]. Instead of using the 2M unbalanced training set, we train on a (slightly more) balanced subset consisting of 500K samples (AS-500K). Details are provided in the appendix. Because each sample has multiple labels, we train with a binary cross-entropy (BCE) loss and report mean average precision (mAP) over all classes, following standard practice.
100 [12] consists of egocentric videos capturing daily kitchen activities. The dataset consists of 90,000 variable length clips spanning 100 hours. We report results for action recognition following standard protocol [12] each action label is a combination of a verb and noun, and we predict both using a single network with two 'heads', both trained with a cross-entropy loss. The top scoring verb and action pair predicted by the network are used, and Top-1 action accuracy is the primary metric. Actions are mainly short-term (average length is 2.6s with minimum length 0.25s). VGGSound [10] contains almost 200K video clips of length 10s, annotated with 309 sound classes consisting of human actions, sound-emitting objects and human-object interactions. Unlike AudioSet, the sound source for each clip is 'visually present' in the video. This was ensured during dataset creation through the use of image classifiers. After filtering clips that are no longer available on YouTube, we end up with 172,427 training and 14,448 test clips. We train with a standard cross- entropy loss for classification and report Top-1 and Top-5 classification accuracy.
4.2 Implementation details
Our backbone architecture follows that of ViT [16] identically, specifically we ViT-Base (ViT-B, I. = =1, Nu = 12, d = 3072)' initialised from ImageNet-21 [14], however we note that our method is agnostic transformer backbone. Unless otherwise specialised, we use B bottleneck tokens for all experiments with bottleneck fusion. Bottleneck tokens are initialized using a Gaussian with mean of and standard deviation of 0.02, similar to the positional embeddings in the public ViT [16] code. We randomly sample clips of t seconds for training. RGB frames for all datasets are extracted at 25 fps. For AudioSet and VGGSound we sample RGB frames over the sampling window of length t with a uniform stride of length (t x 25)/8. We extract 16 x 16 patches from each frame of size 224 x 224, giving us a total of 14 x 14 1568 patches per video. For (because the segments are shorter), we sample 32 frames with stride 1. Audio for all datasets is sampled at 16kHz and converted to mono channel. Similar to [23], we extract log mel spectrograms with a frequency dimension of 128 computed using a 25ms Hamming window with hop length 10ms. This gives us an input of size 128 100r for seconds of audio. Spectrogram patches are extracted with size 16 x 16, giving us 50 x 8 = 400 patches for 8 seconds of audio. For images we apply the standard data augmentations used in [6] (random crop, flip, colour jitter), and for spectrograms we use Spemeugment [45] with max time mask length of 192 frames and max frequency mask length of 48 bins following AST [23]. We set the base learning rate to 0.5 and train for 50 epochs, using Mixup [59] with a 0.3 and stochastic depth regularisation [27] with probability = 0.3. All models (across datasets) are trained with a batch size of 64, synchronous SGD with momentum of 0.9, and cosine learning rate schedule with warmup of 2.5 epochs on TPU accelerators using the Scenic library [13].
Inference: Following standard practice, uniformly sample multiple temporal crops from the clip and average per-view logits to obtain the final result. The number of test crops is set to 4.
4.3 Ablation analysis
In this section we investigate the impact of the different architectural choices in MBT. Unless otherwise specified, we use the mini-AudioSet split for training and report results on the AudioSet eval split. More ablations on backbone size and pretraining initalisation can be found in the appendix.
L is the number of transformer layers, Nu is the number of self-attention heads with hidden dimension d.

---

4.3.1 Fusion strategies
We implement all the three fusion strategies described in Sec. 3.2:
(i Vanilla self-attention Unrestricted pairwise attention between all latent units within a layer;
(ii) Vanilla cross-attention with separate weights: Same above, but we now have separate weights for each modality. The latent units are updated via pairwise attention with all other latent units from both modalities; and finally
(ii)) Bottleneck fusion: Here all cross-modal attention must pass through bottleneck fusion latents. Note that these three fusion strategies only describe attention flow between tokens within layer. For strategies (ii) and (iii), we also conduct experiments showing the impact of restricting cross-modal attention to layers after a fixed fusion layer ,,. We investigate models with different fusion layers,
L, 0, 2, 4, 6, 8, 10, 12, and present the results in Fig. 3.2
Sharing weights for both modalities: We first investigate the impact of sharing the encoder weights for both modalities (strategy (i) vs (ii)). The results can be found in Fig. 1 in the appendix. When modalities are fused at earlier layers, using separate encoders improves performance. For models with later fusion layers, performance is similar for both models. We hence use separate modality weights for further experiments.
Fusion layer: We then investigate the impact of varying the fusion layer Ly, for the latter two strategies: (ii) Vanilla Cross-Attention and (iii) Bottleneck Fusion. We conduct experiments with L = 0, 4, 6, 8, 10, 12. We fix the input span t to 4s and the number of bottleneck tokens B to 4. We conduct 3 runs for each experiment and report mean and std deviation. As can be seen from Fig. (left). 'mid fusion' outperforms both early (Ly- = 0) and late fusion (L, = 12), with optimal performance obtained by using fusion layer L = 10 for vanilla cross- aotsenttnnn and L5 8 for bottleneck attention. This suggests that the model benefits from restricting mool connections to later layers, allowing earlier layers to specialise to learning unimodal features, however still benefits from multiple layers of cross-modal information flow. In appendix D, we confirm that mid fusion outperforms late fusion across number of different datasets.
A Attention bottlenecks: In Fig. 3, we also examine the effect of bottleneck attention vs vanilla cross-attention for multimodal fusion. We find that for all values of L5 restricting flow to bottlenecks improves or maintains performance, with improvements more prominent at lower values of Ly. At LJ 10, both perform similarly, note that at this stage we only have 3 fusion layers in the model. Our best performing model uses attention bottlenecks with Ly = 8, and we fix this for all further experiments. We also compare the amount of computation, measured in GFL.Ps, for both fusion strategies (Fig. 3, right). Using a small number of bottleneck tokens (in our experiments B = 4) adds negligible extra computation over late fusion model, with computation remaining largely constant with varying fusion layer I.y. This is in contrast to vanilla cross- cross-fusion, which has a non-negligible computational cost for every layer it is applied to. We note that for early fusion (L,- = 0), bottleneck fusion outperforms vanilla tross-attention by over 2 mAP, with less than half the computational cost. Number of bottleneck tokens B: We experiment with B 4, 36, 64, 256 and 1024, and find that performance is relatively consistent (all within 0.5 mAP). We hence fix the number of tokens to B 4 for all experiments. It is interesting that with such a small number of cross-modal connections through only 4 hidden units (B- = 4) at each cross- moosl layer, we get large performance gains over late fusion (Fig. 3), highlighting the importance of allowing cross-mode modal information to flow at multiple layers of the model.
4.3.2 Input sampling and dataset size
In this section we investigate the impact of different modality sampling strategies. We also compare to single modality baselines the visual-only and audio-only baselines consist of vanilla transformer model applied to only the RGB or spectrogram patches respectively.
Sampling window size t: An advantage of our transformer based model is that we can easily input variable length token sequences. We experiment with varying the sampling window t with the following values 2, 4, and 8 seconds (note that all videos in AudioSet are 10s), and show results in Fig. 4- At inference, we uniformly sample multiple windows covering the entire video. While the number of spectrogram patches N. changes with t, we keep the number of RGB patches N
NNote that L 12 refers to late fusion, where logits are only aggregated after the classifiers, and neither fusion strategy (ii) nor (iii) is applied, but we show results on the same plot for convenience. "Averaged over 3 runs. Because error bars are small in the plot we also provide them in Table 3 in the appendix.
7

---

Attention Bottlenecks
+ Vanillaa Cross-Attention
200
GFLOPS
MAP
150
100
8
10
12
2
10
12
Fusion Layer L,
Fusion Layer Ly
Figure 3: The impact of using attention bottlenecks for fusion on performance (left) and compute (right) at different fusion layers T. AudioSet, using clip span / = 4 and B 4 bottleneck tokens. Attention bottlenecks improve performance lower computational cost.
Visual-only
Audio-only
Audio-visual
- Visual-only
Audio-only
Audio visual
45
35
mWp
40
35
30
Video clip span (s)
Number of Training Samples
le5
Figure 4: The effect of varying input clip span t on the AudioSet test set.
Figure 5: The effect of training data size on the AudioSet test set.
fixed by changing the stride of frames (to avoid rumning out of memory). Our results indicate that the performance of both the audio and audio-visual fusion model increases with input span, however the performance of the visual-only model slightly decreases (we hypothesize that this due to the increased fixed stride, meaning fewer frames are randomly sampled during training). We fix = 8s in all further experiments.
Synchronous asynchronous sampling: Given that auditory and visual events may not always be perfected aligned in videos [32], we also investigate asynchronous sampling of different modalities. Here input windows are sampled independently from the entire video clip for each modality, Results are provided in Fig. 2 in the appendix. We find performance to be largely robust to either case, and use synchronised sampling for all further experiments.
so for simplicity
Modality MixUp: While applying Mixup regularization [59] to training, we note that there are two different ways apply for multimodal inputs the standard approach
sample of mixup weights from a Beta distribution using the parameter o, and use it to generate all virtual modality- pairs [59]. We also explore modified version which we call modality mixup, which samples an independent weight for each modality. Modality mixup imposes stronger augmentation than standard mixup. leading to slight improvement (42.6 mAP to 43.9 mAP) on AudioSet.
Impact of dataset size: We show the impact of varying the number of training samples in Fig. 5, and find monotonic increase with dataset size (more steeply for audio-only than visual-only).
4.4 Results
Comparison to single modality performance: We compare MBT to visual-only and audio-only baselines on AudioSet (Table 1), Sspic-Kitchens (Table and VGGSound (Table 3). Note we use the best parameters obtained via the ablations above, i.e. bottleneck fusion with t 8, B = 4, F = 8 and modality mixup. For all datasets, multimodal fusion outperforms the modality baseline, demonstrating the value of complementary information. The relative importance of modalities for the classification labels varies (audio-only has higher relative performance for AudioSet and lower for Epic- KitcheK,chhns, while both audio and visual baselines are equally strong for VGGSound). This is (unsurprisingly) largely a function of the dataset annotation procedure and positions VGGSound uniquely suitable dataset for fusion. We also show that audio-visual fusion
single

---

Model
Training Set
A only
only
AV Fusion
GBlend [56]
MiniAS
29.1
22.1
37.8
GBlend [56]
FullAS-2M
32.4
18.8
41.8
Attn Audio-Visue [18]
FullAS-2M
38.4
25.7
46.2
Perceiver [29]
FullAS-2M
38.4
25.8
44.2
MBT
MiniAS
31.3
27.7
43.9
MBT
AS-500K
44.3
32.3
52.1
Table 1: Comparison to SOTA on AudioSet [21]. We report mean average precision (mAP). We
outperform works that train on the full Audioset (2M samples), while we train on only 500K samples.
Model
Modalities
Verb
Noun
Action
Damen al. [12]
42.1
21.5
14.8
AudioSlowFast [33]
A
46.5
22.78
15.4
TSN [55]
V,
60.2
46.0
33.2
TRN [60]
V,
65.9
45.4
35.3
TBN [32]
A,
V.F
66.0
47.2
36.7
TSM [40]
V,
F
67.9
49.0
38.3
SlowFast [19]
65.6
50.0
38.5
MBT
A
44.3
22.4
13.0
MBT
62.0
56.4
40.7
MBT
A, V
64.8
58.0
43.4
Table 2: Comparison to SOTA on EpicKitchens-100 [12]. Modalities are A: Audio, V: Visual, F:
Optical flow. fUses pretraining on VGGSound.
provides slight performance gains for traditionally video only datasets such Kinetics and Moments in Time (details provided in Appendix C ). We also examine per-class performance on the Audioset dataset (Figures 3 and in the Appendix), and find that for the top 60 classes (ranked by overall performance), audio-visual fusion improves performance over audio only or visual only for almost all (57 out of 60) classes, except for 'bagpiping', 'emergency vehicle' and 'didgeridoo' which have strong audio signatures. For classes such as "bicycle' and 'shufiling cards' where audio signals are weaker, fusion improves over the audio-only baseline by over 60% absolute AP.
Comparison to state of the art: We compare MBT to previous fusion methods on AudioSet in previous works on fusion (even though we only train on quarter of the
Table We outperform training set - 500K samples), including the recently introduced Perceiver [29] which uses early fusion followed by multiple self attention layers, and Attn Audio Visual [18] which uses self-attentin fusion on top of individual modality CNNs. We compare to previous video classification methods on Epic-K Epie-Kitehens Table 2, and note that our model outperforms previous works that use vision only, as well as TBN [32] which uses three modalities RGB, audio and optical flow. Given VGGSound is
Model
Modalities
Top-1 Ace
Top-5 Acc
Chen et alt [10]
A
48.8
76.5
AudioSlowFastt [33]
A
50.1
77.9
MBT
A
52.3
78.1
MBT
V
51.2
72.6
MBT
A.V
64.1
85.6
Table 3: Comparison to the state of the art on VGGSound [10]. Modalities are A: Audio, V:
Visual, F: Optical flow. & We calculate metrics on our test for fair comparison using the scores provided by the authors.

---

Mid Frame
Vanilla Fusion
MBT
Mid Frame
Vanilla Fusion
MBT
Baby
String instrument
vodeling
Figure 6: Attention Maps. We compute maps of the attention from the output CLS tokens to the RGB image input space for vanilla self-attention model and MBT on the Audioset test For each video clip, we show the original middle frame on the left with the ground truth labels overlayed at the bottom. The attention particularly focused on sound source regions in the video that contain motion, eg. the fingertips on the piano, the hands on the string instrument, faces of humans. The bottlenecks in MBT further force the attention be localised to smaller regions of the images (i.e the mouth of the baby on the top left and the mouth of the woman singing on the bottom right).
a relatively new dataset, we compare to two existing audio-only works* (Table 3), and set the first aware of) this dataset.
audiovisual benchmark (that Visualisation of attention maps Finally, we compute maps of the attention from the output CLS tokens to the RGB image input space using Attention Rollout [1]. Results on test images for both vanilla fusion model and MBT trained on Audioset-mini (fusion layer L = 8) are shown in Figure 6. We show the attention maps summed over all the frames the video clip. We note that first, the model focuses on semantically salient regions in the video for audio classification, particularly regions where there motion that creates or modifies sound, i.e. the mouth of humans making sounds, fingertips on piano. hands and instruments. This is unlike state of the art sound source localisation techniques trained with images [9], which tend highlight the entire object. We further note that the attention maps for MBT are more localised to these regions, showing that the tight bottlenecks do force the model focus only on the image patches that are actually relevant for the audio classification task and which benefit from early fusion with audio.
Conclusion
We propose transformer architecture (MBT) for audiovisual fusion, and explore number of different fusion strategies using cross-atie attsentitonoonn between latent tokens. We propose a novel strategy to restrict cross-modal attention via small set of fusion 'bottlenecks', and demonstrate that this improves performance over vanilla cross-attention at lower computational cost, achieving state the art results number of henchmarks. Future work will involve extending MBT to other modalities such as text and optical flow.
Limitations: The fusion layer is hyperparameter and may need to be tuned specifically for different tasks and datasets. We also only explore fully supervised fusion, and future work will tackle extensions to a self-supervised learning framework.
Broader impact: Multimodal fusion strategies are important for machine learning, as fusing complementary information from different modalities can increase robustness when applied real world applications. We also note that transformers are in general compute-heavy, which can have adverse environmental effects. We propose token fusion method via bottlenecks that helps reduce computational complexity when applying transformers for multimodal fusion. Finally, we observe that training datasets contain biases that may render models trained on them unsuitable for certain applications. It is thus possible that people use classification models (intentionally or not) to make decisions that impact different groups in society differently, and it important to keep this in mind
"To fairly compare to these works, we obtain the scores on the full VGGSound test set from the authors, and compute accuracy metrics our slightly smaller test described in Sec. 4.1.
10

---

when deploying, analysing and building upon these models.
Acknowledgements: We would like to thank Joao Carreira for helpful discussions on the Perceiver [29].
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928, 2020.
[2] Hassan Akbari, Linagzhe Yuan, Rui Qian. Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 2021.
[3] Jean-Bage Saan-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. In NewrlPS, 2020.
[4] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
[5] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In 2018.
[6] Anurag Amab, Mostafa Dehghami, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. ICCV, 2021.
[7] Yusuf Aytar, Carl Vondrick. and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurIPS, 2016.
[8] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. ICCV, 2021.
[9] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, 2021.
[10] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. VGGSound: A large-scale audio-visual dataset. In ICASSP, 2020.
[11] Tsuhan Chen and Ram R Rao. Audio-visune visual integration in multimodal communication. Pro- ceedings of the IEEE, 86(5):837885, 1998.
[12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision. arXiv preprint 2xXiv:2006.13250 2020.
[13] Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Tay. Scenic: A JAX library for computer vision research and beyond. arXiv preprint arXhv:2110.11400 2021.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. Icee, 2009.
[15] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[17] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: speaker-indepeeddet audio-visual model for speech separation. ACM Transactions on Graphics (TOG), 37(,):1-11, 2018.

---

[18] Haytham M Fayek and Anurag Kumar. Large scale audiovisual learning of sounds with weakly labeled data, arXiv preprint arXiv:2006.0159) 01595, 2020.
[19] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 1200-6211 2019.
[20] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, volume 5. Springer, 2020.
[21] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and for audio events. In ICASSP. pages 776-780. IEFF, 2017.
dataset
[22] Bernard Ghanem, Juan Carlos Niebles, Cees Snoek, Fabian Caba Heilbron, Humam Alwassel, Victor Escorcia, Ranjay Krishna, Shyamal Buch, and Cuong Duc Dao. The activitynet large- scale activity recognition challenge 2018 summary. arXiv preprint arXiv: / 1808.0366,, 2018.
[23] Yuan Gong, Yu-An Chung, and James Glass. AST: audio spectrogram transformer. arXiv preprint arXiv:2104.01778, 2021.
[24] David Harwath, Antonio Torralba, and James R Glass. Unsupervised learning of spoken language with visual context. NeurIPS, 2017.
[25] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean- Sann-Bptiste Alayrac, and Aida Ne- matzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. arXiv preprint arXiv: 102.00099, 2021.
[26] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. CNN architectures for large-scale audio classification. In ICASSP, pages 131-135. IEEE, 2017.
[27] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[28] Vladimir lashin and Esa Rahtu. Multi-modal dense video captioning. In CVPR Workshops, pages 958-959, 2020.
[29] Andrew Jaegle, Felix reira. Perceiver: General perception with iterative attention. arXiv preprint arXiv:2103.03220 2021.
Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Car-
[30] Aren Jansen, Daniel PW Ellis, Shawn Hershey, Channing Moore, Manoj Plakal, Ashok C Popat, and Rif A Saurous. Coincidence, categorization, and consolidation: Learning to recognize sounds with minimal supervision. In ICASSP, 2020.
[31] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 170k: 069500 2017.
[32] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio- visual temporal binding for egocentric action recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5492-5501, 2019.
[33] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Slow-fast auditory streams for audio recognition. In ICASSP, pages 855-859. IEEE, 2021.
[34] Yelin Kim, Honglak Lee, and Emily Mower Provost. Deep learning for robust feature generation in audiovisual emotion recognition. In ICASSP. IEEE, 2013.
[35] Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Pa- rameter efficient multimodal transformers for video representation learning. arXiv preprint arXiv:2012.04122 042.244, 2020.
[36] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. Entangled transformer for image captioning. In ICCV, pages 8928-8937, 2019.
12

---

[37] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li. Learning to generate diverse dance motions with transformer. arXiv preprint Satv::2000.08177// 2020.
[38] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 17X 2019.
[39] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa, Learn to dance with aist++: Music conditioned 3d dance generation. arXiv preprint arXiv:2101.08779, 2021.
[40] Lin, Chuang Gan, and Song Han. Temporal shift module for efficient video understanding.
2019 iece. In ICCV, pages 7082-7092, 2019.
[41] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for disionaadd-languag tasks. In NewrlPS, 2019.
[42] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):502- 508, 2019.
[43] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML, 2011.
[44] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisen- sory features. In ECCV, 2018.
[45] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoe Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1904v:9904.08779, 2019.
[46] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. In ICCV, 2021.
[47] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE Signal Processing Magazine, 34(6))96-108, 2017.
[48] Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmenta- tion for environmental sound classification. IEEE Signal Processing Letters, 2493)277--28,, 2017.
[49] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. Look before you speak: Visually contextualized utterances. In CVPR. 2021.
[50] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-29, 2005.
[51] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05744 2019.
[52] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019.
[53] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds. arXiv preprint arXiv:2011.0114 2020.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 11t.:1770003622. 2017.
[55] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. ECCV. Springer, 2016.
13

---

[56] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In CVPR. pages 12695-12705, 2020.
[57] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020.
[58] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In CVPR, 2019.
[59] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0%. 2017.
[60] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba, Temporal relational reasoning in videos. In ECCV, pages 803-818, 2018.
14


Reviews:
[{"review_id": "wQq3OZSFlAa", "reviewer": "Reviewer_SoXm", "summary": "This paper conducts an experimental study of different fusion methods for multimodal audiovisual transformers, and proposes a new fusion mechanism. They mix ordinary transformer layers with new fusion attention layers. These new layers perform cross-attention, in which one modality is used and the other is used as the key/values. This takes places using special fusion tokens, which have a function similar to the classification tokens in ordinary transformers. They find that (1) mid-fusion can obtain better results than early/late fusion in both their model and \"vanilla\" fusion, (2) attention bottlenecks improve performance, (3) they outperform the very recent, state-of-the-art Perceiver model on AudioSet classification. They also conduct additional experiments on VGG-Sound and Epic Kitchens, in which they compare with other state-of-art models. The supplement contains several other datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n- The attention bottleneck mechanism is simple and has a compelling motivation over \"vanilla\" fusion.\n- The idea of examining the effectiveness of early/late fusion has not, as far as I know, been examined in the context of transformer models.\n- This paper looks at fusion designs in a more systematic way than most prior work, which is enabled by the simple design of their transformer architecture.\n\nNegatives:\n- The cross-attention function MCA() is seemingly asymmetric (L122), using only X as the query in the multi-headed attention. Are both directions of attention used in Eq. 8 and Eq. 9, thereby allowing by the bottleneck and the modalities to be updated?\n- The comparison to Perceiver is not quite apples-to-apples, due to the use of more extensive augmentation (as well as the different train set). Though it might not be essential, it would clarify the comparison if a version of Perceiver were trained using the same setup.\n- There is no direct comparison to AVSlowFast [49], despite training on similar datasets (such as Kinetics-400 in the supplement), and having similar goals.\n\nOverall:\nThis paper presents a very interesting analysis of different fusion strategies in transformers. The bottleneck idea is simple and compelling. The model obtains large gains from fusion on a variety of datasets. I therefore lean towards acceptance.\n\nAfter the discussion:\n- The rebuttal clarified my question about the asymmetric updates.\n- The response about comparisons seems reasonable.\nOverall, I still lean towards acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These were addressed adequately in the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "rcafjmqVW2N", "reviewer": "Reviewer_TS5J", "summary": "For efficient multimodal fusion, the authors proposed the attention bottlenecks for audio-visual classification tasks. Compared to previous works, they emphasize their bottleneck architecture \"to collate and condense the most relevant information in each modality.\" They showed that it also has the advantage of a lower computational cost. They evaluate the proposed method using three competitive methods, vanilla self-attention, vanilla cross-attention, and their attention bottlenecks. They noted that the position of modality fusion was a critical factor among early, mid and late. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- They showed that they can \"keep the number of bottleneck tokens in the network to be much smaller than the total number of latent units per modality.\" I believe this is a novel observation and very interesting. \n- I appreciate the explicit paragraph for the limitations. \n\n\n### Clarity\n\nc1) L111, could you more elaborate on the functionality of the special token $z_{cls}$ for readers why it is especially needed?\n\nc2) L122, \"MCA\" is appeared without definition.\n\nc3) L130, which interval of the audio waveform is converted to a single spectrogram? Could you clarify specifically?\n\nc4) L193, what do you mean by a \"slightly more\" balanced subset used? Slightly more than 500k samples? Could you provide the details?\n\nc5) L222, how do you get \"80 x 5 = 400\" patches for 8 seconds?\n\nc6) Footnote 2, what do you mean by \"... and neither strategy (ii) nor (iii) is applied\"? (what is the subject and to what?)\n\nc7) L296, for the modality mixup, how do you choose the weight for the labels, as in the Mixup? The explanation for the modality mixup seems to be insufficient to understand correctly.\n\n\n### Minors\n\nm1) \"All (section) headings should be lower case except for the first word and proper nouns\" according to the official style guideline.\n\nm2) L56, \"a baselines\" -> \"baselines\"\n\nm3) L217, \"8 x 14 x 14\" -> \"8 x 16 x 16\", since you extract 16 x 16 patches from each frame?\n\n----\nI appreciate your thorough feedbacks. After reading other reviews, I cannot resist to increase my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "L1) I suggest updating the title, \".. for Multimodal Fusion\" since this work limits to bi-modal and audio-visual classification tasks as the authors mentioned in the Conclusion. They fail to show how to scale to more than two modality tasks and if it is also effective other than audio-visual classification tasks.\n\nL2) In the Abstract, the authors argued that the bottleneck forces \"to collate and condense the most relevant information in each modality and only share what is necessary.\" Since the authors evaluate the method using macro metrics for the task, I suggest toning down to \"most relevant\" -> \"relevant\" and \"only share\" -> \"share\". The current explanation seems to be too strong without sufficient evidence.\n\nL3) Although the attention bottlenecks have the advantage of computational cost, this is sensitive to the fusion layer $L_f$. In some cases, the proposed method, attention bottlenecks are indifferent from vanilla cross-attention as shown in Figure 3. This means if we adjust $L$ for the smaller or larger dataset, the hyperparameter $L_f$ would be critical to its performance. The authors omit this aspect in the Limitations of the Conclusion section.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "iaK09RFKr1u", "reviewer": "Reviewer_x2ZW", "summary": "The authors propose a new Multimodal Bottleneck Transformer (MBT) for audio-visual learning. Taking image and spectrogram patches as inputs, MBT uses self and cross-attention operations to model both unimodal and cross-modal information at multiple layers in the network. Notably, it adopts a small set of ‘bottleneck’ latent units to force the model to collate and condense the most relevant information in each modality and only share what is necessary. Extensive ablation studies and experiments are conducted on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens, and VGGSound.\n\n***Post-Rebuttal***\n\nI've read comments from fellow reviewers and the rebuttal from the authors.\n\nThank the authors for considering my comments and responding to my questions. The rebuttal successfully addressed my major concerns. Thus, I would like to keep my initial positive rating. \n\nThe authors should add the missing details, discussion, and new results in the rebuttal to the paper. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\n+ The proposed model is new and interesting. Two distinct modalities are integrated by a unified multimodal transformer architecture: MBT.  It uses self and cross-attention mechanisms to model both unimodal and cross-modal information at multiple layers. With the bottleneck units, the network becomes more effective and efficient. \n\n+ Thorough ablation studies and experiments on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens, and VGGSound are provided.  Ablation studies can validate the effectiveness of the fusion design and the proposed method obtains strong results on the benchmarks.  \n\nOverall, I like the proposed method and appreciate the thorough experiments. But, I still have some questions and hope the authors can clarify them. \n\nCons: \n\n- A single visual scene is usually grouped together and has strong local patterns, and multiple visual scenes are usually non-overlapping. I think that is one of the main reasons why convolution, patch-based nonlocal/transformer networks work. However, different from images, a single sound source contains diverse isolated time-frequency patterns; different audio sources can occupy the same frequency at the same time; a single audio source can cover almost all frequency ranges at a time (dog barking sound).  Thus, I worry that these isolated patches might not be able to fully reflect the correct semantic of sound sources. Could the authors provide more discussion and analysis on the validity of audio transformers? Are there any failure examples due to using spectrogram patches? \n\n- To reduce the computation complexity of pairwise attention, fusion bottleneck tokens are introduced. However, it is not clear to me how to initialize these fusion bottleneck tokens. Only the updating method is provided in the paper.\n\n- The authors note that bottlenecks can force the model to collate and condense the most relevant information in each modality and only share what is necessary. However, without any visualizations, the statement is not fully convincing. The authors should provide some examples (e.g., visualizing attention weights) to visually demonstrate the capacity of the bottleneck attention. \n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discussed the limitations and Broader Impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ZKQ5uzqgFkL", "reviewer": "Reviewer_yUDq", "summary": "The authors propose a new transformer architecture MBT. The MBT uses additional learnable tokens to summarize multimodal information between different modalities and an attention modification to process the constructed multimodal sequences efficiently. They perform ablations on fusion strategies and comparison with related models on audio-visual datasets (such as AudioSet, Epic Kitchen 100, VGGSound). They show improved results on most benchmarks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The Proposed MBT model shows improved results on most of the benchmarks. However, there is a missing baseline. ViViT [1] can perform better on Epic Kitchens 100 as a video only model. Thus additional experiments are required to show that it improves state-of-the-art and it is worth an additional compute. \n\n  - ViViT-L/16x2 Fact. Encoder outperform MBT using only Video on Verb (66.4 > 64.8 (A,V) > 62. (V)), Action (44 > 43.4 (A, V) > 40.7 (V)). The ViViT is only worse versus MBT (A,V) on Noun (56.8 < 58.0) but MBT's visual (V) version is a bit worse (56.8 > 56.4).\n  - ViViT-B/16x2 Fact Encoder does on action better (43.7 > 43.4 (A, V) > 40.7 (V))\n  - [1] Arnab, Anurag, et al. \"Vivit: A video vision transformer.\" *arXiv preprint arXiv:2103.15691* (2021).\n\n- There are only two modal video-audio experiments. It is not clear how this idea is versatile to different multimodal scenarios (e.g., any combination of Vision, Audio, Text, Flow; coarse and fine information).\n\n### Minor\n\n- There is no attempt to understand the bottleneck token in terms of multimodal relations.\n\n- Multimodal cross attention (MCA) in section 3.2.2 is related to Co-Attention previously shown by the following work [2]. \n\n  - [2] Hendricks, Lisa Anne, et al. \"Decoupling the role of data, attention, and losses in multimodal transformers.\" *arXiv preprint arXiv:2102.00529* (2021).\n\n- Is the order to update the bottleneck tokens essential for performance in section 3.2.3 (e.g., with visual or audio first)? Have you explored other ways how to utilize bottlenecks tokens? Currently, the updates are factorized and asymmetric.\n\n- The tables with comparison do not show backbone architecture, or the number of parameters, or FLOPs. It is hard to understand trade-offs.\n\n- There is no backbone scaling study or transfer learning experiments.\n\n### Originality:\n\n- The idea of multimodal fusion via attention bottlenecks tokens is somewhat novel. It is an adaptation of a classification token to a multimodal application. The modified multimodal attention module is novel.\n\n### Quality: \n\n- Overall it is good work. Specifically, the ablation studies are mostly extensive.\n\n### Clarity: \n\n- The manuscript is well written. Tables with comparison can be improved (see Minor).\n\n### Significance:\n\n- The results are promising. However, a unimodal video model such as ViViT can achieve better results on Epic Kitchens 100. Thus it is not clear that it is state-of-the-art and additional experiments are needed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously recommend an acceptance. They acknowledged that the proposed attention bottleneck module is simple and demonstrated to be effective by extensive experiments and ablation analyses. They also appreciated an empirical exploration of different fusion mechanisms in Transformer models. The initial reviews raised some concerns about insufficient experiments and asked clarifying questions, and the rebuttal successfully cleared up the questions. Overall, this is a nice paper addressing an important problem of modeling multimodal data and proposes a simple method based on Transformer architectures, which is a timely topic for this conference.
Decision: Accept (Poster)