Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper conducts an experimental study of different fusion methods for multimodal audiovisual transformers, and proposes a new fusion mechanism. They mix ordinary transformer layers with new fusion attention layers. These new layers perform cross-attention, in which one modality is used and the other is used as the key/values. This takes places using special fusion tokens, which have a function similar to the classification tokens in ordinary transformers. They find that (1) mid-fusion can obtain better results than early/late fusion in both their model and "vanilla" fusion, (2) attention bottlenecks improve performance, (3) they outperform the very recent, state-of-the-art Perceiver model on AudioSet classification. They also conduct additional experiments on VGG-Sound and Epic Kitchens, in which they compare with other state-of-art models. The supplement contains several other datasets.
main_review: Positives:
- The attention bottleneck mechanism is simple and has a compelling motivation over "vanilla" fusion.
- The idea of examining the effectiveness of early/late fusion has not, as far as I know, been examined in the context of transformer models.
- This paper looks at fusion designs in a more systematic way than most prior work, which is enabled by the simple design of their transformer architecture.

Negatives:
- The cross-attention function MCA() is seemingly asymmetric (L122), using only X as the query in the multi-headed attention. Are both directions of attention used in Eq. 8 and Eq. 9, thereby allowing by the bottleneck and the modalities to be updated?
- The comparison to Perceiver is not quite apples-to-apples, due to the use of more extensive augmentation (as well as the different train set). Though it might not be essential, it would clarify the comparison if a version of Perceiver were trained using the same setup.
- There is no direct comparison to AVSlowFast [49], despite training on similar datasets (such as Kinetics-400 in the supplement), and having similar goals.

Overall:
This paper presents a very interesting analysis of different fusion strategies in transformers. The bottleneck idea is simple and compelling. The model obtains large gains from fusion on a variety of datasets. I therefore lean towards acceptance.

After the discussion:
- The rebuttal clarified my question about the asymmetric updates.
- The response about comparisons seems reasonable.
Overall, I still lean towards acceptance.
limitations_and_societal_impact: These were addressed adequately in the conclusion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a new Multimodal Bottleneck Transformer (MBT) for audio-visual learning. Taking image and spectrogram patches as inputs, MBT uses self and cross-attention operations to model both unimodal and cross-modal information at multiple layers in the network. Notably, it adopts a small set of ‘bottleneck’ latent units to force the model to collate and condense the most relevant information in each modality and only share what is necessary. Extensive ablation studies and experiments are conducted on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens, and VGGSound.

***Post-Rebuttal***

I've read comments from fellow reviewers and the rebuttal from the authors.

Thank the authors for considering my comments and responding to my questions. The rebuttal successfully addressed my major concerns. Thus, I would like to keep my initial positive rating. 

The authors should add the missing details, discussion, and new results in the rebuttal to the paper. 
main_review: Pros: 

+ The proposed model is new and interesting. Two distinct modalities are integrated by a unified multimodal transformer architecture: MBT.  It uses self and cross-attention mechanisms to model both unimodal and cross-modal information at multiple layers. With the bottleneck units, the network becomes more effective and efficient. 

+ Thorough ablation studies and experiments on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens, and VGGSound are provided.  Ablation studies can validate the effectiveness of the fusion design and the proposed method obtains strong results on the benchmarks.  

Overall, I like the proposed method and appreciate the thorough experiments. But, I still have some questions and hope the authors can clarify them. 

Cons: 

- A single visual scene is usually grouped together and has strong local patterns, and multiple visual scenes are usually non-overlapping. I think that is one of the main reasons why convolution, patch-based nonlocal/transformer networks work. However, different from images, a single sound source contains diverse isolated time-frequency patterns; different audio sources can occupy the same frequency at the same time; a single audio source can cover almost all frequency ranges at a time (dog barking sound).  Thus, I worry that these isolated patches might not be able to fully reflect the correct semantic of sound sources. Could the authors provide more discussion and analysis on the validity of audio transformers? Are there any failure examples due to using spectrogram patches? 

- To reduce the computation complexity of pairwise attention, fusion bottleneck tokens are introduced. However, it is not clear to me how to initialize these fusion bottleneck tokens. Only the updating method is provided in the paper.

- The authors note that bottlenecks can force the model to collate and condense the most relevant information in each modality and only share what is necessary. However, without any visualizations, the statement is not fully convincing. The authors should provide some examples (e.g., visualizing attention weights) to visually demonstrate the capacity of the bottleneck attention. 





limitations_and_societal_impact: Yes, the authors discussed the limitations and Broader Impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a new transformer architecture MBT. The MBT uses additional learnable tokens to summarize multimodal information between different modalities and an attention modification to process the constructed multimodal sequences efficiently. They perform ablations on fusion strategies and comparison with related models on audio-visual datasets (such as AudioSet, Epic Kitchen 100, VGGSound). They show improved results on most benchmarks. 
main_review: - The Proposed MBT model shows improved results on most of the benchmarks. However, there is a missing baseline. ViViT [1] can perform better on Epic Kitchens 100 as a video only model. Thus additional experiments are required to show that it improves state-of-the-art and it is worth an additional compute. 

  - ViViT-L/16x2 Fact. Encoder outperform MBT using only Video on Verb (66.4 > 64.8 (A,V) > 62. (V)), Action (44 > 43.4 (A, V) > 40.7 (V)). The ViViT is only worse versus MBT (A,V) on Noun (56.8 < 58.0) but MBT's visual (V) version is a bit worse (56.8 > 56.4).
  - ViViT-B/16x2 Fact Encoder does on action better (43.7 > 43.4 (A, V) > 40.7 (V))
  - [1] Arnab, Anurag, et al. "Vivit: A video vision transformer." *arXiv preprint arXiv:2103.15691* (2021).

- There are only two modal video-audio experiments. It is not clear how this idea is versatile to different multimodal scenarios (e.g., any combination of Vision, Audio, Text, Flow; coarse and fine information).

### Minor

- There is no attempt to understand the bottleneck token in terms of multimodal relations.

- Multimodal cross attention (MCA) in section 3.2.2 is related to Co-Attention previously shown by the following work [2]. 

  - [2] Hendricks, Lisa Anne, et al. "Decoupling the role of data, attention, and losses in multimodal transformers." *arXiv preprint arXiv:2102.00529* (2021).

- Is the order to update the bottleneck tokens essential for performance in section 3.2.3 (e.g., with visual or audio first)? Have you explored other ways how to utilize bottlenecks tokens? Currently, the updates are factorized and asymmetric.

- The tables with comparison do not show backbone architecture, or the number of parameters, or FLOPs. It is hard to understand trade-offs.

- There is no backbone scaling study or transfer learning experiments.

### Originality:

- The idea of multimodal fusion via attention bottlenecks tokens is somewhat novel. It is an adaptation of a classification token to a multimodal application. The modified multimodal attention module is novel.

### Quality: 

- Overall it is good work. Specifically, the ablation studies are mostly extensive.

### Clarity: 

- The manuscript is well written. Tables with comparison can be improved (see Minor).

### Significance:

- The results are promising. However, a unimodal video model such as ViViT can achieve better results on Epic Kitchens 100. Thus it is not clear that it is state-of-the-art and additional experiments are needed.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: For efficient multimodal fusion, the authors proposed the attention bottlenecks for audio-visual classification tasks. Compared to previous works, they emphasize their bottleneck architecture "to collate and condense the most relevant information in each modality." They showed that it also has the advantage of a lower computational cost. They evaluate the proposed method using three competitive methods, vanilla self-attention, vanilla cross-attention, and their attention bottlenecks. They noted that the position of modality fusion was a critical factor among early, mid and late. 
main_review: - They showed that they can "keep the number of bottleneck tokens in the network to be much smaller than the total number of latent units per modality." I believe this is a novel observation and very interesting. 
- I appreciate the explicit paragraph for the limitations. 


### Clarity

c1) L111, could you more elaborate on the functionality of the special token $z_{cls}$ for readers why it is especially needed?

c2) L122, "MCA" is appeared without definition.

c3) L130, which interval of the audio waveform is converted to a single spectrogram? Could you clarify specifically?

c4) L193, what do you mean by a "slightly more" balanced subset used? Slightly more than 500k samples? Could you provide the details?

c5) L222, how do you get "80 x 5 = 400" patches for 8 seconds?

c6) Footnote 2, what do you mean by "... and neither strategy (ii) nor (iii) is applied"? (what is the subject and to what?)

c7) L296, for the modality mixup, how do you choose the weight for the labels, as in the Mixup? The explanation for the modality mixup seems to be insufficient to understand correctly.


### Minors

m1) "All (section) headings should be lower case except for the first word and proper nouns" according to the official style guideline.

m2) L56, "a baselines" -> "baselines"

m3) L217, "8 x 14 x 14" -> "8 x 16 x 16", since you extract 16 x 16 patches from each frame?

----
I appreciate your thorough feedbacks. After reading other reviews, I cannot resist to increase my score to 7.
limitations_and_societal_impact: L1) I suggest updating the title, ".. for Multimodal Fusion" since this work limits to bi-modal and audio-visual classification tasks as the authors mentioned in the Conclusion. They fail to show how to scale to more than two modality tasks and if it is also effective other than audio-visual classification tasks.

L2) In the Abstract, the authors argued that the bottleneck forces "to collate and condense the most relevant information in each modality and only share what is necessary." Since the authors evaluate the method using macro metrics for the task, I suggest toning down to "most relevant" -> "relevant" and "only share" -> "share". The current explanation seems to be too strong without sufficient evidence.

L3) Although the attention bottlenecks have the advantage of computational cost, this is sensitive to the fusion layer $L_f$. In some cases, the proposed method, attention bottlenecks are indifferent from vanilla cross-attention as shown in Figure 3. This means if we adjust $L$ for the smaller or larger dataset, the hyperparameter $L_f$ would be critical to its performance. The authors omit this aspect in the Limitations of the Conclusion section.
review_title: null

Meta-review: The reviewers unanimously recommend an acceptance. They acknowledged that the proposed attention bottleneck module is simple and demonstrated to be effective by extensive experiments and ablation analyses. They also appreciated an empirical exploration of different fusion mechanisms in Transformer models. The initial reviews raised some concerns about insufficient experiments and asked clarifying questions, and the rebuttal successfully cleared up the questions. Overall, this is a nice paper addressing an important problem of modeling multimodal data and proposes a simple method based on Transformer architectures, which is a timely topic for this conference.
Decision: Accept (Poster)