Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes TriBERT to handle the modalities of RGB image, poses and audios for tackling the task of sound source separation. The BERT model is used to extract features following ViLBERT. Experiments show appealing results on sound separation metrics compared with the author-reproduced previous SOTA methods.
main_review: This paper illustrates a relatively reasonable story in its introduction, however, the story cannot be fully supported by the experiments conducted. The usage of the BERT model in the field of visual sound separation is quite interesting. The model descriptions are mostly clear enough for reproducibility. The reviewer thinks this paper is of certain value to the field of visual sound separation.

** The strengths of this paper are summarized as follows:

++ This paper is well written and easy to follow.

++ The idea of involving BERT in visual sound separation is interesting.

++ Most of the model design is reasonable and with care, which is important for application papers.

** The weaknesses of the paper are as follows:

1. The conducted experiments do not match with the story in the introduction and the title.

a) The title is “Human-centric” audio-visual representation learning, however, the experiments are conducted on MUSIC21 dataset with sound source separation task. I am not saying that separating instrument sound is not entirely “human-centric”, but this is only a relatively small area in the  “human-centric” audio-visual field. I am expecting to see experiments conducted on human speech and its relationship with maybe co-speech gestures when I read the introduction. The experiments cannot support the title or the introduction at all. 

b) The core of ViLBERT or related work, is to use proxy tasks for training and show the effectiveness of learned features with downstream applications, while this paper basically focuses on sound separation. Although this is easy to understand: there are not so many downstream tasks if the model is trained on MUSIC dataset.

The reviewer suggests that the authors either focus on visual sound separation (with a title like “TriBERT for visual sound separation”) or conduct experiments on large-scale “human-centric” datasets such as TED talks or AudioSet, and show the effectiveness of representation learning on more downstream tasks.

2. No qualitative results and no video supplementary shown. Demo videos are extremely important for audio-visual papers. With no video results, it is impossible for reviewers to know the true performance of the method. 

Also, the authors do not provide any visual results of visual segmentation while the visual branch is designed for this. 
The retrieval experiments are weird. Though I understand the authors are trying to match the “representation learning” in the title, these kinds of experiments are useless and have never been conducted in related studies.

3. The quantitative experiments are not persuasive. Although our community cares less about computational cost, the paper lacks a lot of ablation studies. For example, this paper leverages attention U-Net as backbone, which is rarely used in previous works. The entire visual network is more complicated than all previous work. And the three modalities in the TriBERT are not evaluated efficiently. What if one of the modality is erased? More needed ablations are not listed.

4. Details. Why use only three frames which is the same as PixelPlayer for the pose branch? This is not reasonable given that the pose frames should cover more temporal information as performed in Music Gesture.

Why Table1 is below Table2 and why evaluate on MUSIC alone given that it is a subset of MUSIC21? The settings are not reasonable. 

-------------------------------------

The rebuttal addresses some of my concerns and provides positive revision directions. However, it requires a major revision from the current form (the story needs to change, or large amounts of experiments should be shown), thus I still do not recommend acceptance of this paper. I raise my rating from 4 to 5 and encourage the authors to keep perfecting this paper.
limitations_and_societal_impact: The authors have addressed their limitations. But these are major weaknesses of their work that could not be simply regarded as limitations. The authors should consider changing their story. 

Video results should be included to show the qualitative results, so that more limitations could be revealed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: By pointing out the recent transformer-based models are mostly designed for visual-language data, this paper introduce TriBERT which specifically targeting on audio-visual modalities to address such limitation. Inspired by ViLBERT, TriBERT is a transformer based model which enables contextual feature learning across three modalities: vision, pose and audio with the use of flexible co-attention. Besides, this paper introduce a learned visual tokenization scheme based on spatial attention and leverage weak-supervision to allow granular cross-modal interactions for visual and pose modalities. To demonstrate the proposed approach, TriBERT is pretrained on MUSIC221 and show improved performance in audio-visual sound source separation and other audio-visual-pose retrieval. 
main_review: This paper introduces TriBERT, a three-stream model which can accept three modalities, i.e. audio, visual, pose. By utilizing tri-modal co-attention blocks, generic audio-visual representations are achieved for audio-visual tasks of sound source separation and audio-visual-pose retrieval. This work of extending the usage of transformer-based models for more modalities other than just visual and language, i.e. audio-visual, is well motivated. 

However, my main concern is the novelty of this work is limited. Though, to extending transformer to audio and pose is not a trivial problem, the approach been used is straightforward. The whole TriBERT model is kind of a combination of some existing building blocks, e.g. BERT, graph CNN, segmentation net, Vggish net, UNet, etc. The training objective is designed as a multi-task manner, i.e. visual and pose classification, audio classification, and multi-modal sound source separation. From both aspects of model design, and algorithm/training objective, I do see much novelty appears. 
Besides, the experiments are not strong enough. First, with comparing with only a few SOTA approaches,  I can not be fully convinced. Also, why there no comparisons with sota for retrieval task? 
For many recent audio-visual representation learning work, the approaches are usually demonstrated on 2-3 audio-visual tasks, e.g. audio event classification, and activity classification. I am wondering would these tasks can be considered to demonstrate the authors' claim of 'learning generic audio-visual representation'?
I also did not see any ablation studies of the proposed approach. For example, the authors claim they proposed a new learned visual tokenization scheme, but no experimental results are shown to demonstrate the effectiveness of this tokenization scheme.
 
limitations_and_societal_impact: Some of my suggestions:
1) add more SOTA approaches for comparison on sound source localization task
2) add SOTA comparisons on retrieval task
3) add ablation studies of the proposed approach, e.g. comparisons of different visual tokenization approaches
4) further demonstrate the claim of 'learning generic audio-visual representations', e.g. consider tasks of audio/video classification
5) comparing with other audio-visual work, pose is the an extra modalities and the unique part in this paper. I would suggest to add some experiments and analysis on the contribution of adding pose
6) no code was submitted. I am wondering the reproducibility of this work
7) minor: alter positions of  Table 1 & 2
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The goal of this work is to learn human-centric audio-visual representations. The authors claim to be first in using transformer models in the context of granular audio-visual detection or segmentation tasks (e.g. sound source separation). The authors propose TriBERT, which is a three-stream model with tri-modal attention to generate generic representaitons for AV tasks. The three modalities are audio, video and pose (keypoints). The model is trained with weak supervision (video level labels). The model gives state-of-the-art results on sound source separation.
main_review: The paper introduces BERT-like contextualized representation learning for the audio-visual domain. This is first in the context of AV source separation to my knowledge. The method appears to be effective for the task. 

The architecture is based on existing work (VilBERT), but adds additional components such as segmentation network, spectrogram prediction mask and pose input.

The results are strong on the datasets shown. However, as the authors noted, the experiments are limited to only two datasets, one of which cannot be reproduced completely due to missing files (line 307). Ablations are also not given to justify design choices.

The results would be more convincing if the results could be reproduced for other human-centric datasets such as action recognition (UCF101) or audio-visual speech recognition (LRS3). 

Due to the limitations in experimental validation, I recommend borderline reject.

The English is fine, but the paper is not easy to follow (perhaps due to the large number of components to explain).


Additional questions:

Does the authors have an explanation as to why V+A>P is so much worse than A>P in top1 accuracy in Table 3?

Isn't the pose just a processed form of the visual input? Is it just to make it more explicit so that it is easier for the network to process?

What information is fed into the network in the case of occlusion?
limitations_and_societal_impact: Addressed in Section 5
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents TriBERT a BERT-like model for learning multi-model representations of three modalities simultaneously, namely: vision, pose and audio. The architecture of TriBERT takes inspiration from the visual-linguistic model VilBERT (notably known for the applications in VQA) in terms of its Transformer architecture and of its multi-modal attention (the latter is extended in the paper to 3 modalities instead of 2).

TriBERT training comprises four losses to optimize. In particular, after the projection of the 3 unimodal feature sequences into the joint feature space by the multimodal transformer, there are 3 sequences of features, namely: <h_v>, <h_p> and <h_a> for respectively enriched visual, pose and audio features. Then, the <h_v> and <h_p> features are (independently) trained to localize and predict the sound source (the musical instrument(s) which is (are) played in the recording) using 2 (independent) BCE losses. It should be noted that the localization part is learnt in a weakly-supervised manner as only global instrument classification annotations are available in the training data. The <h_a> audio features are trained to classify the sound source (the 3rd BCE loss).

Finally, the 4th supervision task is the multi-modal sound source separation one. The original audio spectrogram is mixed with other audio spectrogram(s) and the task consists in predicting a spectrogram mask which would separate the original audio from the others (the 4th BCE loss). Following [37], for this last task, a U-Net neural architecture is used. U-Net takes the mixed audio spectrogram at its input and the fusion of <h_v>, <h_p> and <h_a> at its bottleneck representation to predict the desired spectrogram mask.

Once trained, TriBERT is evaluated on the MUSIC21 dataset for the sound separation task, and it also fine-tuned and evaluated for the multi-modal retrieval downstream task.

main_review: *Originality*

The paper has a primary original contribution: in particular, to the best of my knowledge, it is the 1st study to report a conclusive approach for training of a tri-modal BERT-like model on visual, audio and pose modalities with impressive results on such a complicate downstream task as multi-modal retrieval.

The paper has also two other minor contributions. Firstly, it demonstrates that unimodal visual features can be learnt from scratch in an end-2-end manner (while a common approach is to extract object features with a pretrained object detector such as Faster-RCNN as it is done in VilBERT, for example), and secondly, they illustrate that sound source prediction, separation and localization are powerful supervision tasks which are enough to train a complex Transformer architecture with relatively little annotations (only the tags of played instruments).

*Quality*

I believe that the authors have properly cited and compared their approach to all significant related work, most notable to [15] and [55]. Moreover, the approach of [15] has been re-implemented and retrained in order to use the same unimodal features for the pose modality (the ones learnt with a graph CNN on keypoints extracted with the AlphaPose toolbox).

All in all, in my opinion, the paper is technically sound.

*Clarity*

The paper is very well written and easy to follow. The authors commit to publishing the source code upon the paper acceptance.

*Typos*

1.	Line 93: “of of” - > “of”.

*Significance*

To my knowledge, this is the 1st paper which has successfully managed to learn joint representations of vision, pose and audio inputs with a single BERT-like architecture. Moreover, only very shallow annotations (the musical instrument type) of the training dataset (MUSIC21) have been used for training. Given the convincing experimental results on the sound separation task where the strong Sound-of-Pixels [55] and MUSIC-Gesture [15] baselines have been largely outperformed, as well as the multi-modal retrieval results reported in Table 3 which demonstrate that TriBERT managed to learn a joint feature space for the three modalities, I believe that this paper will be useful for the community as a 1st example of what might become a new standard in audio-visual processing (as it was the case for VilBERT in the visual-linguistic domain). Therefore, I am rather inclined to recommend the acceptance of this paper.

At the same time, I would like to highlight the following limitations and suggestions / questions for the authors:

1.	The experiments are done only on a single dataset (MUSIC is actually a subset of MUSIC21). The results would have been even more convincing, had there been another dataset for evaluation.

2.	I would be interested to see an ablation on the contributions of each modality to the final sound separation results reported in Tables 1 and 2. Thus, I believe that the pose and acoustic modalities are much more important than the visual one for this task. Does the visual modality really contribute in the final score?

3.	It would also be informative to see some qualitative results on the sound localization given the fact that this task has been learnt in a weakly supervised manner (without explicit annotations on the localization).

4.	Finally, how can you explain that the retrieval Top-1 accuracy for the pose modality given only audio modality is higher than given both audio and visual modalities (Table 3)?


limitations_and_societal_impact: I do not see any negative social impact for this research.
review_title: null

Meta-review: This work proposes a new transformer-based model that can learn across three modalities: vision, pose, and audio for tackling the task of sound source separation. The authors did a good job during rebuttal and turned two slightly negative reviewers into positive ones.  The final score is very borderline (three borderline accepts and one borderline reject).  This work could clearly benefit from a more extensive experimental evaluation, but AC feels this work is very interesting and deserves to be published on NeurIPS. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are also encouraged to make other necessary changes.
Decision: Accept (Poster)