Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a transformer-based architecture for learning representations from video-audio-text triplet data without manual data annotation. The paper studies modality-specific transformers (one transformer for one modality) and modality-agnostic transformers (a shared transformer for all modalities). The learned representation is evaluated on various downstream tasks on image, video, audio and video-text.
main_review: The originality of the paper is limited. The standard VATT model is identical to [1] (Multimodal versatile network, Alayract et al.) except changing the ConvNet backbones into transformers. The novel point is the modality-agnostic shared transformer for all three modalities. The proposed DropToken is a useful technique to reduce the computation overhead for transformers. 

The paper shows a large number of experimental results, especially for downstream tasks. The overall quality is good.

The paper is clearly written. 

The significance of the results is moderate:
1. The key method of the paper is a transformer-based architecture for multimodal learning. In detail, it uses identical architecture and loss design (sec 3.3, 3.4) except replacing the backbone with transformers. But when comparing with [1] (Table 4 and supp Table 2), the proposed transformer backbone underperforms or gives similar results as its ConvNet counterpart, also the discussion is limited on this point.
2. The downstream performance of modality-agnostic transformers is not far from modality-specific transformers, which is encouraging for future works. 
3. Table 5 shows the proposed DropToken is useful for reducing computation while keeping the downstream performance.


[1] Self-supervised multimodal versatile network, Alayrac et al.

limitations_and_societal_impact: As stated above, one of the key contributions is a transformer-based multimodal architecture, however it does not outperform the ConvNet counterpart when using the same loss. Discussion on this point would be valuable. 

The proposed modality-agnostic transformer is novel, and the paper shows its performance on downstream tasks. But it’s still not clear to me what can be captured in that unified transformer. I am curious to see more qualitative results like the attention maps per encoder layer for each modality.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission is about multi-modal representation learning from unlabeled data (self-sup.) using Transformer. This work deals with video, audio, and text data as multi-modal data. The approach to tackle representation learning is based on Transformer + contrastive learning.
The contrastive learning in this work is interesting in that 1) noisy association is considered in multiple instances learning way [64], and 2) the different level of semantic granularity is carefully considered.
Also, the authors adopt a simple DropToken to improve efficiency.

The training requires very large-scale training using HowTo100M and AudioSet. The authors demonstrated its effectiveness on action recognition, audio event classification, zero-shot video retrieval, image classification.

Multi-modal data is encoded by independent feature extractors, which is a small difference from other Vision Transformers. Naturally, the authors extend the positional encoding to 3D positional encoding (horizontal, vertical, and temporal axes) with relativeness.

Overall, the technical contribution is a bit weak, but this submission contains interesting analyses that are worthwhile to report in the community and also achieve state-of-the-art in downstream tasks.
main_review: 
Pros
- Trendy topic with good results
- Pointing out different levels of semantic granularity
- Proposing DropToken to improve training efficiency
- Interesting experiment about the ImageNet finetuning results with the model pre-training video
- Extensive evaluation

Cons
- Weak technical contribution, and company-level large-scale training requirement
- Composition of existing developments

<Comments>

This work could be categorized as an analysis-oriented paper.

* Technical contribution: This work can be viewed as a similar line of research to the contrastive learning-based Vision Transfomer representation learning, e.g., [18]. In this regard, this work also can be viewed as a straightforward extension to multi-modal-based self-supervised data. The consideration of semantic granularity difference and Droptoken is only different from the previous work, but this reviewer feels that technical contribution is not significant. 

* Semantic granularity: This reviewer concurs with the authors' argument on the different levels of semantic granularity in multi-modal learning. This reviewer thinks this must have been considered in all multi-modal learning research. At least, good to see the emphasis on it in this work.

* Showing a possibility of the existence of a unified model in a multi-modal regime. Nice.


** Question: Considering the out-of-context works on the recent MLP based backbones, what would be the true benefit of Transformer in this work? This reviewer does not see whether Transformer is a truly important component in the multi-modal learning regime like this work, other than large capacity. 
Also, the submission does not include much about the necessity or motivation of the Transformer in multi-modal learning, but just mentioning no one did yet.\
What is the motivation of Transformer in the authors' opinion?

It would be more thorough to consider the motivation of the Transformer in this context.


======================== After rebuttal ========================

The authors have addressed most of the comments. However, the concerns on the technical contribution and the position of this submission are not adequately addressed, which are shared across the majority of the reviewers.

For this reason, I downgrade the score. 
This reviewer still thinks that the benefits of accepting this work would be slightly higher than that of rejecting it to the community.


limitations_and_societal_impact: The authors discuseed the requirement of large computation, and model bias due to the use of uncurated multi-modal videos, which are reasonble concerns.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper concerns multimodal self-supervised learning. It adopts the now ubiquitous contrastive learning framework to learn video, audio, text representation from each raw signal in joint embedding spaces. The proposed architecture heavily relies on MMV [1], with a twist on Visual/Audio Transformer encoders and modality-agnostic Transformers. Comprehensive experimental results are conducted on four diverse video(-text) tasks across ten datasets.
main_review: This paper is sending some mixed messages. On the one hand, it demonstrates Transformers' decent capability on learning from raw visual/audio signals, in the context of multimodal self-supervised learning. Also, the possibility of unifying encoder across modalities through a modality-agnostic framework. On the other hand, this unique capability is at the cost of larger models, more training resources, and sometimes performance degradation.

Since the proposed methods **heavily** rely on the previous work MMV [1], missing this important baseline in many result tables is a major flaw (despite that MMV is not originally evaluated on these highly relevant datasets) and hinders a fair comparison between CNN-based methods and Transformer-based methods. On the few datasets that have MMV results reported (e.g., Tab. 4, Tab. 2 in Supp.), the proposed methods overall underperform the baseline, sometimes significantly, despite using much larger models (2-4x) and computation resources (over 2x TPU days). This puts a doubt on the effectiveness of the proposed method.

Besides, the reported model variants are inconsistent throughout the paper (e.g., in Tab. 2 and 3, VATT-Base while in Tab. 4 VATT-Medium). In Tab. 1, columns indicating #params, #training data, modality should be added for better comparison.
limitations_and_societal_impact: Appear to be.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper provides a framework for self-supervised multimodal representations from unlabeled data with Transformer backbones. The authors study two styles of Transformer encoders: modality-specific or modality-agnostic.  Finally, they achieve good performance on various downstream tasks including action recognition, audio event classification, text-to-video retrieval and image classification.
main_review: - This paper is well written and the topic of this paper is valuable.

- The novelty of this paper might be limited. The framework of self-supervised multimodal representation learning is almost as same as MMV [1] except replacing CNN backbones with Transformers.  DropToken technique does not show many advantages especially over training with lower resolution (79.9 acc. with 548.1 GFLOPS v.s. 80.2 acc. with 572.5 GFLOPS in Table 6). 

- It is interesting to study the modality-agnostic encoder and the results are encouraging. However, it is not the main part of this paper. It would be nice to take a more detailed analysis and prove that a modality-agnostic encoder can achieve competitive or better performance than a modality-specific one.

- The performance comparisons with previous multimodal representation learning methods (MIL-NCE, MMV) are missing. Authors should following previous papers that evaluating on UCF and HMDB and report the results.

- The detail of DropToken is confusing. Is the DropToken used in both the training and testing phases? Is it used when transferring to downstream tasks or only during pre-training?

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a self-supervised pure transformer-based multimodal representation learning including video-audio-text, i.e., VATT. VATT consists of  linear projection of input for each modality, modality-specific or agnostic transformer encoder, and multimodal projection head. VATT uses NCE loss for video-audio feature learning and MIL-NCE loss for text-video features. Also, the authors proposed DropToken as an efficient method for training. 
They employ HowTo1M and AudioSet datasets for pretraining and evaluate their VATT for diverse finetuning tasks including video action recognition, audio event classification, and zero-shot text-to-video retrieval. With intensive experiments and ablation studies, they provide promising results and constructive analysis. 

main_review: ### Strength
#### Originality
- It is novel to use audio for multimodal pretraining in addition to video and text.
- Pure transformer architecture-based video-audio-text learning is novel considering NeurIPS submission deadline. 

#### Quality & Significance
- This paper is well-organized to easy to understand.
- Multimodal pretraining is a challenging but significant topic. 

#### Clarity & Experiments
- Experiments are extensive and the results look promising on diverse downstream tasks.
- Analysis including feature visualization is helpful.

### Weakness
Although I like this paper, there are some rooms to be improved.
- Some core previous studies was missed in related work such as VideoBERT [Sun et al. 2019]. Also, The authors need to compare VideoBERT. Also, in related work, it is helpful to enhance the story by including recent studies on self-supervised learning from uncurated data in computer vision domains such as SEER [Goyal et al. 2021] and image-based self-supervised learning for videos [Feichtenhofer et al. 2021].
- Why did the author use word-level linear projection as text input instead of BPE tokenization, which is de facto in training LM? Please clarify this reason. 
- The author employs multi-level common space projection with hierarchy. Is there any performance drop using pair-wise feature loss with plat-style? If then, how significant?
- For negative pairs, how about making matching inter video clips? for example, frames from video A and audio from video B. 
- As the authors described in the paper, VATT model sizes are variable according to each modality encoder size and whether the encoder is modality-agnostic or not. At least, table 1, 2 and, 3 need to include model parameter sizes. 
- In table 1, What data are used as the pretraining or scratch-training for other baseline models? It need to be clarified. 
- How is the performance on Kinect-700? Including MMV, many recent work evaluated their model on Kinect-700. 
- For DropToken, how is the effect of DropToken on each modality? Did the authors try to apply DropToken to a specific modality?
- Why VATT-MBS was compared instead of VATT-Base, unlike other tables? Considering the parameter size, I think that VATT-Base with the same batch-size and epochs is more suitable for comparison. 
- How is the performance when increasing batch-size and epochs similar to MMV or MIL-NCE in table 4? As the authors argued the importance of batch-size and epoch, the results on large batch size and epochs with the same model will help to clarify the argument. 
- Readability can be improved. For example, VATT-Base means VATT-BBS? In Table 4, "VATT-MBS" is used instead of "VATT-Medium". Consistent term usages are very helpful. Overall, the captions of Tables need to be self-contained with more explanation and term definition. Also, it is better to clarify the use of self-supervised ViT (79.9%) for better understanding. 

#### References
- [Sun et al. 2019] A Joint Model for Video and Language Representation Learning. ICCV 2019.
- [Goyal et al. 2021] Self-supervised Pretraining of Visual Features in the Wild. arXiv 2021.
- [Feichtenhofer et al. 2021] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning. CVPR 2021.

limitations_and_societal_impact: No special societal impact.
review_title: null

Meta-review: The merits and weaknesses of this submission were discussed at length in back-and-forth conversations with the authors. The overall sense is that, although the technical novelty of the submission is limited, the paper provides a valuable large-scale empirical study on sharing a single transformer backbone architecture across different modalities. Several reviewers comment on the importance of sharing such results with the community. The ACs find that this is particularly relevant given that few labs are equipped with the resources to carry out such a study (thus here the ACs dissent from the opposite point made by some reviewers). The submission also introduces DropToken, a simple strategy to significantly reduce the computational cost of training large-capacity transformers. Based on these contributions, the ACs agree to accept this submission but encourage the authors to incorporate the feedback provided by reviewers, especially as it concerns the motivation for using transformers and the novelty claims. 
Decision: Accept (Poster)