Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a vision-language transformer representation learning model, denoted as TVLT. The proposed model is based on masked auto-encoders comprised of two encoders (one for vision and one for audio) and a single shared decoder. 
The model was trained to predict the masked segments together with newly proposed Vision-Audio Matching (VAM) contrastive loss. The authors provide results for several tasks namely: Audio-to-Video Retrieval, Multimodal Emotion Analysis, Audio-to-Image Retrieval, and Speech-based Visual Question Answering. 
strengths_and_weaknesses: Strengths: 
1) The authors provide results for several tasks, demonstrating the effectiveness of the proposed approach. 
2) The authors provide a detailed analysis of their method together with an ablation study. 

Weaknesses: 
1) Novelty is limited. The proposed method is very similar to the masked auto-encoder which was recently proposed for image, where the authors additionally include another audio encoder getting the audio spectrogram "image" as another input. 
2) Missing references to previous work.
3) Experimental description can be improved.
questions: 1) The main issue with this submission is its novelty. By nature audio is pretty different than image, however, the authors used the same encoder arch for both domains where they consider the spectrogram as another image. Did the authors explore other encoders as well? 
2) Considering the synthetic speech data, I'm a bit surprised that synthesized speech from the text was more beneficial than just text considering the way this speech was generated. In other words, when speech was synthesized from the text it will contain the content and artificial prosodic feature, and I'm not sure why should it be better than just text? Can the authors provide more details about this process?
3) There are some missing references the authors should pay attention to, especially when considering the textless approach: 
* Shi, Bowen, et al. "Learning audio-visual speech representation by masked multimodal cluster prediction." arXiv preprint arXiv:2201.02184 (2022).
* Owens, Andrew, and Alexei A. Efros. "Audio-visual scene analysis with self-supervised multisensory features." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
* Surís, Didac, et al. "Cross-modal embeddings for video and audio retrieval." Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018.
* Lakhotia, Kushal, et al. "On generative spoken language modeling from raw audio." Transactions of the Association for Computational Linguistics 9 (2021): 1336-1354.
* Kreuk, Felix, et al. "Textless speech emotion conversion using decomposed and discrete representations." arXiv preprint arXiv:2111.07402 (2021).
* Kharitonov, Eugene, et al. "Text-free prosody-aware generative spoken language modeling." arXiv preprint arXiv:2109.03264 (2021).
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a transformer that takes both speech and images as input to learn a joint vision and language representation. The transformer is trained with two tasks, one is masked autoencoding and the other is whether an image matches an audio clip. The models are then fine tuned on a particular task, for example, audio-to-video retrieval, multimodal sentiment classification, and visual question answering with speech as input. Experiments show that the approach performs well compared to the state of the art.
strengths_and_weaknesses: The paper is easy to follow. The proposed architecture is simple, yet performs well. The ablation study and comparison of various input modalities are convincing.
questions: > Section 3.2

It is a bit unclear what architecture is used for the decoder. It is also unclear how a decoder could be used to generate both images and spectrogram patches. Images have 3 channels, while a spectrogram only has 1. How is this achieved exactly?

> ... we construct half of the visio-audio pairs inside a batch as mismatched (negative) pairs (x^{V+}, x^A), by replacing video frames x^{V+} ...

Should it be (x^{V-}, x^A)?

> Section 5.3

It's unclear in general how the model is used for each downstream task. Are there any output layers added for each task? Are both the images and spectrograms present during testing? The model in principle should have the ability to take only a single modality instead of both.
limitations: The societal impact is inherited from the vision-language tasks.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an audio-visual multi-modal representation learning model called TVLT, and it shows the effectiveness of the learned representations on the downstream tasks that require the understanding of visual and linguistic information. Unlike the previous vision and language representation learning methods that take text input to help elicit the linguistic information, it takes raw audio input instead of the text input but still achieves comparable results with much more improved inference efficiency. In experiments, TVLT shows comparable performance to the text-based counterpart model on audio-to-video retrieval and video-based multimodal sentiment analysis. Although TLVT is slightly worse than the text-based counterpart model, it is still able to achieve comparable results while being 50x faster during inference. In addition, this paper conducts extensive experiments such as a comparison to the SOTA textless models and an ablation study on masking strategy and various model architectures.
strengths_and_weaknesses: ### Strengths

- The paper is well-written and the flow of the writing is natural and neat so it was easy to read.
- The proposed method is original and it presents an interesting research direction for future works.
- I believe this work would give a significant impact on this field and its simple architecture is likely to be used by other people.
- Especially, the minimalist design of TVLT allows to scale up the model so it is expected to give another possibility of large-scale multi-modal model as GPT-3 did in NLP.

### Suggestions

- As far as I know, the tables should be centered in the papers submitted to NeurIPS so I think it should be fixed.
- There are several suggestions of the experiments that can be conducted in the future work. I listed them in the questions section below.
- In line 128, I think it might be a typo, $(x^{V+} → x^{V-})$
- It can give more insights if there are samples of the masked data instances.
questions: * Is it possible to use each of the representations for uni-modal subtasks (e.g. Can audio representations be used for ASR training)?
* What if the text-based model uses all the modalities; text, image, and audio? This is because, as it is mentioned in the paper, the audio modality contains more information that is absent in the text.

(I totally understand that it is almost impossible to conduct every experiment, but it is written somewhat as part of curiosity. So don't take these questions too negatively :D)
limitations: This paper adequately addressed the limitations and did considerations about the negative societal impact.
review_title: null

Meta-review: metareview: Reviewers acknowledge that the proposed method is simple and performs well and have the potential to present an interesting research direction for future works. The authors respond actively to the review comments and one reviewer raises the score after author response. It is a good paper, and I recommend acceptance. Authors should follow the reviewer's suggestions to update the paper to address some the questions. 
confidence: Certain
Decision: Accept