TriBERT: Human-centric Audio-visual Representation Learning
Authors: Tanzila Rahman
Mengyu Yang
Leonid Sigal
Venue: NeurIPS 2021 Poster (2021)
Keywords: BERT
Transformer
multi-modal
joint representation

Abstract: The recent success of transformer models in language, such as BERT, has motivated the use of such architectures for multi-modal feature learning and tasks. However, most multi-modal variants (e.g., ViLBERT) have limited themselves to visual-linguistic data. Relatively few have explored its use in audio-visual modalities, and none, to our knowledge, illustrate them in the context of granular audio-visual detection or segmentation tasks such as sound source separation and localization. In this work, we introduce TriBERT -- a transformer-based architecture, inspired by ViLBERT, which enables contextual feature learning across three modalities: vision, pose, and audio, with the use of flexible co-attention. The use of pose keypoints is inspired by recent works that illustrate that such representations can significantly boost performance in many audio-visual scenarios where often one or more persons are responsible for the sound explicitly (e.g., talking) or implicitly (e.g., sound produced as a function of human manipulating an object). From a technical perspective, as part of the TriBERT architecture, we introduce a learned visual tokenization scheme based on spatial attention and leverage weak-supervision to allow granular cross-modal interactions for visual and pose modalities. Further, we supplement learning with sound-source separation loss formulated across all three streams. We pre-train our model on the large MUSIC21 dataset and demonstrate improved performance in audio-visual sound source separation on that dataset as well as other datasets through fine-tuning. In addition, we show that the learned TriBERT representations are generic and significantly improve performance on other audio-visual tasks such as cross-modal audio-visual-pose retrieval by as much as 66.7% in top-1 accuracy. 

TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation
Tanzila Rahman 1.3
Mengyu Yang
Leonid Sigal
University of British Columbia
University of Toronto
3 Vector Institute for Al
*Canada CIFAR AT Chair
trahman&&cs ubc. ca,
my- my- sy-yang@mail wang@maii..toroontooo... utoronto.ca. ca, lsigalèes. latgalles.uuc....
Abstract
The recent success of transformer models in language, such BERT, has motivated the use of such architectures for multi-modal feature learning and tasks. However, most multi-modal variants (e.g., ViLBERT) have limited themselves to visual- linguistic data. Relatively few have explored its use in audio-visual modalities, and none, our knowledge, illustrate them in the context of granular audio-visual detection or segmentation tasks such as sound source separation and localization. In this work, we introduce TriBRRT transformer- based architecture, inspired by ViLBERT, which enables contextual feature learning across three modalities: vision, pose, and audio, with the use of flexible co-attention. The use of pose keypoints inspired by recent works that illustrate that such representations can significantly boost performance many audio-visual scenarios where often one or more persons are responsible for the sound explicitly (e.g., talking) implicitly (e.g., sound produced as a function of human manipulating an object). From a technical perspective, as part of the TriBERT architecture, tokenization scheme based on spatial attention and leverage weak-supervision to allow granular cross-modal interactions for visual and pose modalities. Further, we supplement learning with cound-source separation loss formulated across all three streams. We pre-train our model the large MUSIC21 dataset and demonstrate improved performance in audio-visual sound source separation on that dataset as well other datasets through fine-tuning. In addition, show that the learned TriBERT representations are generic and significantly improve performance on other audio-visual tasks such cross-modal audio-visual-pose retrieval by as much as 66.7% in top-1 accuracy,
introduce learned visual
Introduction
Multi-modal audio-visual learning (57. which explores and leverages the relationship between visual and auditory modalities, has started emerge
important sub-field of machine learning and
computer vision. Examples of typical tasks include: audio-visual separation and localization, where the goal is segment sounds produced by individual objects in audio and/or to localize those objects in a visual scene [15 T6 42 (5): and audio-visual correspondence, where the goal is often 53). Notably, some of the most recent audio-visual methods (T5) leverage human pose keypoints, or landmarks, as an intermediate or contextual representation. This tends to improve the overall performance of sound separation, as pose and motion are important cues for characterising both the type of instrument being played and, potentially, over time, the rhythm of the individual piece [5]. It can also serve as an intermediate representation when generating video from acoustic signals g for example.
audio-visuue visual retrieval 23 47
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Most of the existing architectures tend to extract features from the necessary modalities using pre-trained backbones (e.g., CNNs applied to video frames [55], object regions [[o], and audio spec- trograms; and/or graph CNN for human pose [5] and then construct problem-specific architectures that often utilize simple late fusion for cross-modal integration decoding (e.g. to produce spectro- gram masks [15 16 55 This is contrary to current trends in other multi-modal problem domains, where over the past few years, approaches have largely consolidated around generic multi-modal feature learning architectures that are task agnostic to produce contextualized feature representations and then fine-tune those representations to a variety of tasks (e.g., visual question answering (VQA) or reasoning (VCR)) and datasets. Examples of such architectures include ViLBERT [33], VL-BERT [46, and Unicoder- VL 31. all designed specifically for visual- lingull-inistiic tasks.
Audio-visual representation learning has, in comparison, received much less attention. Most prior works [51] assume a single sound source per video and rely on audio-visual alignment objectives. Exceptions include [39]. which relies on proposal mechanisms and multiple-instance learning [49 or co-clustering 225] These approaches tend to integrate multi-modal features extracted using pre-trained feature extractors (e.g., CNNs) at somewhat shallow level. The very recent variants 6 128 335 leverage transformers for audio-visual representation learning through simple classification 6 and self-supervised (28 or contrastive K5] learning objectives while only illustrating performance on video-level audio-visual action classification. To the best of our knowledge, no audio-visual representation learning approach to date has explored pose one of the constituent modalities; nor has shown that feature integration and contextualization at a hierarchy of levels, as is the case for BERT-like architectures, can lead to improvements granular audio-visual tasks such as audio-visud visual sound source separation.
To address the aforementioned limitations, we formulate human-centric audio-visua representation learning architecture, inspired by ViLBERT [33] and other transformer-base designs, with an explicit goal of improving the state-of-the-art in audio-visual sound source separation. Our transformer model takes three streams of information: video, audio, and (pose) keypoints and co-attends among those three modalities to arrive at enriched representations that can then be used for the final audio- visual sound separation task. We illustrate that these representations are general and also improve performance other auxiliary tasks (e.g., forms of cross- moosl audio-visual pose retrieval). From technical perspective, unlike ViLBERT and others, our model does not rely on global frame-wise features nor an external proposal mechanism. Instead, we leverage a learned attention form visual tokens, akin to [2). and leverage weakly supervised objectives that avoid single sound-source assumptions for learning. In addition, introduce spectrogram mask prediction as one of our pre-training tasks to enable the network to better learn task-specific contextualized features.
Foremost, we introduce tri-modal VilBERT-inspired model, which we call TriBERT, that co-attends among visual, pose keypoint, and audio modalities to produce highly contextualized representations. We show that these representations, obtained by optimizing the model with respect to uni-modal -uueery-uuperiseed) classification and sound separation pretraining objectives, produce features that improve audio-visual sound source separation at large and also work well on other downstream tasks. Further, to avoid reliance on the image proposal mechanisms, we formulate tokenization in the image stream in terms of learned attentional pooling, which is learned jointly. This alleviates the need for externally trained detection mechanisms, such as Faster R- R-CNN and variants. We illustrate competitive performance on a number of granular audio-visual tasks both by using the TriBERT model directly, using it as a feature extractor, or by fine-tuning it.
Contributions:
2 Related works
Audio-visual Tasks. There exists close relationship between visual scenes and the sounds that they produce. This relationship has been leveraged to complete various audio-visual tasks. Based on [57]'s survey of andio-visual deep learning, these tasks can be categorized into four subfields, three of which are addressed in this paper and described in the following three subsections.
Audio-visual Sound Source Separation and Localization. Sound source separation and the related task of sound source localization have been studied quite extensively. Previous works studying separation, also known as the cockiail purty problem [19], leverage multi-modal audio-visual infor- mation [T 14 to help improve performance with respect their audio-only counterparts 26
Examples include learning correlations between optical flow and masked frequencies (9 [3 using

---

graphical models [2I] detecting salient motion signals that correspond to audio events 30 40], and extracting pose keypoints to model human movements [T]] close connection between separation and localization has also been illustrated 40 43
55
56
For example, [6 42 both formulate the
task as one of auditory and visual co-segmentation, either with pre-trained object regions obtained by
the detector [16] or directly from the image 42 All of these approaches contain highly specialized architectures with custom fusion schemes. We aim to leverage the flexibility of transformer models to create generalized multi-modal representations that improve audio-visual tasks.
Audio-visual Representation Learning. The goal is typically to learn aligned representations. The quality of these representations has been shown to greatly impact the overall performance of tasks downstream 4 A common strategy for representation learning is to introduce a proxy task. In the audio-visual space, past works [ 2 38 have trained networks by having them watch and listen to a large amount of unlabeled videos containing both positive samples of matching audio and visual pairs and negative samples of mismatched pairs; the proxy task is binary classification of whether or not the audio and visual match each other. Other proxy tasks include determining whether or not an audio-visual pair is time synchronized [27]: and [29] uses a classification task to identify the correct visual clip or audio stream from set with negative samples. However, these works rely on the assumption that only one main sound source occurs at a time and everything else is background noise. Our model uses a weakly supervised proxy objective to learn representations for multiple sources of sound (two in experiments) occurring simultaneously and also learns to incorporate pose features.
Audio-visual Correspondence Learning. One of the fundamental tasks in correspondence learning related to our work is cross-modality retrieval. Most prior works focus on audio-visual retrieval [24] 36 48 and propose learning a joint embedding space where both modalities can be mapped to. In this space, semantically related embeddings are close to each other and thus retrieval can be performed by selecting the closest embedding to the query from the alternate modality. In our work, we demonstrate that enhanced feature representations obtained by our pretrained model capture aligned semantics and lead to much better cross-modal retrieval than baseline representations.
Visiolinguistic Representation Learning, Our model is inspired by the recent successes of visiolin- guistic representations. Most such approaches leverage a combination of uni-modal and cross- rrdal transformer modules to pre-train generic visiolinguistic representations on masked language and/or multi-modal alignment tasks. For example, K3 proposes separate streams for each modality that communicate with each other through co-attention, while [46] uses a single-stream model that takes both visual and linguistic embeddings as input. our work, we also leverage co-attention modules to learn joint representations between audio, pose, and vision modalities. However, in addition to extending co-attention, we also focus on reformulating image tokenization and demonstrate the ability to learn with weakly-supervise classification objectives as opposed to masked token predictions.
3 Approach
We introduce TriBERT, a network that learns joint representation of three modalities: vision. pose, and audio. We briefly review ViLBERT, the architecture that inspired TriBERT, in Section 3.1 We then describe our TriBERT architecture in Section 3.2 including pretraining tasks and objectives.
3.1 Reviewing Vision-and-Language BERT (ViLBERT)
Motivated by the recent success of the BERT architecture for transfer learning in language modeling, Lu et al. [33] proposed ViLBERT to represent text and visual content jointly. ViLBERT is a two- stream model for image regions and text segments. Each stream is similar to the BERT architecture. containing a series of transformer blocks (TRM) [50). Given an image 1 with corresponding regions-of-interest (RoIs) or bounding boxes Uo, U1, UN and an input sentence S with word tokens Wo, W1, u.,, , the final output representations are hea, hel, ..., huN and ww hei,.... ..., hur for the visual and linguistic features, respectively. To exchange information between the two modalities, the authors introduced a co-attentional transformer layer which computes query (Q), key (K), and value (V) pairs like a standard transformer block. The keys and values from each modality are then fed to the multi-headed attention block of the other modality. The attention block in each stream generates attention-poole features conditioned on the other modality and outputs a multi-modal joint representation which outperforms single-stream models across multiple vision-and -ad--and---aaaan language tasks.
3

---

Raill apenridd
Contlaitt
Flg
hkg
Y
Nuy
Pag
Pgg
Vislon, Pose Audio BERT
508
SMA
805
-MASIO
805
'Pgg Poekee
STPT
Hout Pow harane from Chapit ChiV
Visual Begnertatio Network
SHARCA
BCc
Atlertion Unet
Figure 1: Our TriBERT Architecture. We train TriBERT on the MUSIC21 dataset under two training tasks: (1) Classification and (2) Sound separation. We introduce end-to-end segmentation network for visual embeddings which takes consecutive RGB frames as input and outputs detected object features feed into vision BERT. Following (15), we graph CNN generate pose embeddings as input to pose BERT. For audio, we consider the mixed spectrogram of two audio sources and use a VGGish network to generate audio embeddings feed into audio BERT. We use classification loss to train individual modalities. Due to the lack of annotation for individual objects, we use weakly supervised classification loss to train TriBERT for the vision and pose streams. Following prior works on sound separation, we utilize attention U-net [37], which takes mixed audio spectrogram as input and predicts spectrogram mask guided by aidio-visual-poss features.
3.2 TriBERT Architecture
The architecture of our proposed TriBERT network is illustrated in Figure Inspired by the recent success of ViLBERT in the vision-anddlanguaae domain, we modify its architecture to a three-stream network for vision, pose, and audio. Similar ViLBERT B33, we use bi-directional Transformer encoder [50] as the backbone network. However, TriBERT also introduces integral components that differentiate its architectural design. First, instead of using bounding box visual features generated by
pre- pre-trained object detector [33] or CNN feature columns M7 TriBERT uses jointly trained weakly supervised visual segmentation network. Our end-to-end segmentation network takes sequence of consecutive frames to detect and segment objects, and the corresponding features are pooled and fed as tokens to the visual stream. Second, the pose tokens are characterized by per-person keypoints encoded using Graph CNN, and the audio token is produced by the VGGish Network [22] applied to an audio spectogram. All three types of tokens form the input to TriBERT, which refines them using tri-modal co-attention to arrive at the final multi-modal representations.
Training TrIERRT requires the definition of proxy/pretraining tasks and the corresponding losses (see Section 3.2.1. Specifically, while we adopt token masking used in ViLBERT and others, we are unable to define classification targets per token in our visual and pose streams. This because we only assume per-video labels (e.g., of instruments played) and no access to how those map to attended sounding regions or person instances involved. To address this, we introduce weakly-superviee classification losses for those two streams. Since only one global audio representation is used, this is unnecessary in the audio stream and standard cross- cross-entropy classification can be employed. Finally, motivated by recent works that show that multi-task pretraining is beneficial for ViLBERT [32, introduce an additional spectrogram mask prediction pretraining task which predicts spectrogram masks for each individual audio source from the input spectrogram (bottom block, Figure [.]
Visual Representations. Unlike [33], we consider input video frames instead of detected ob-
ject/bounding box features as our visual input and propose an end-to-end approach to detect and segment objects from each individual frame. Figure illustrates our visual segmentation net-
work which takes in RGB frames as input. To extract global features, we use ResNet50 (20) as the backbone network followed by a 3 x 3 convolution to generate H W visual spatial features which are then fed into the segmentation network. Following [54] use a decou-

---

Segmenbation Network
Expaniis altumtion delector
Attention Map
Drap-out
Drap-out
con-drear
5, I.s
3 Conv
H xix Wy
RGB Frames
Discitiinativ aticotion detector
Top
Average
Pooling
Output Visual Features
Cx1x1
Figure 2:
Visual Segmentation Network. We consider ResNet50 20 followed by 3 3 convolu- tion as our backbone network. It outputs H x W spatial visual feature (V₂) fed into the segmentation network. Class specific attention map (Am) ) is generated by the segmentation network and used to pool top two detected object features. We consider the features as visual embeddings for TriBERT.
pled spatial neural attention structure to detect and localize discriminative objects simultaneously. The attention network has two branches: (1) Expansive atlention detector, which aims to detect object regions and generate the expansive attention map Sg E RCxH*W (top branch of Fig- ure 2 and (2) Discriminative attention detector, which aims to predict discriminative regions and generate the discriminative attention map Sp Xx (bottom branch of Figure 2 The expansive attention detector contains a drop-out layer followed by a x convolution, an- other drop-out layer, a non-linear activation, and spatial- notiaiizaooaaiiiiioonn layer, defined as follows:
(an = F V. Ve(:,,, 5.),
(1)
(2)
where c C C and F(-) denote number of classes and the non-linear activation function, respectively, The final attention map (Am) is generated Am S Sp, where O denotes element-wise multiplication. A spatial average pooling is applied on Am to generate a classification score for each corresponding class and pooled-out top two class features from spatial-visus feature (Ve). The resultant 3 x 2 x 1024 visual embeddings are used to train our proposed TriBERT architecture, where 3 corresponds the number of frames and 2 to the number of "objects" per frame.
Keypoint (pose) Representations. Our goal is to capture human body and finger movement through keypoint representations. Therefore, we extract 26 keypoints for body joints and 21 keypoints for each hand using the AlphaPose toolbox [12]. As a result, we identify the 2D (x, y) coordinates and corresponding confidence scores of 68 body joints. Following (15), we use Graph CNN generate semantic context comprising of those joints. Similar to prior work (52) on action recognition, we construct apatial-Tempora Graph Convolutional Network G [V, E} where each node U2 {V) corresponds to the body joint's 's keypoint and each edge e, E {E} the natural connectivity between those keypoints. We use 2D coordinates of the detected body joints with confidence scores input to each node and construct single frame according to body structure; and (2) connecting each joint with the same joint from the consecutive frames. This way, multiple layers of spatial-temporal graph convolutions are constructed to generate higher-level features for human keypoints. We use publicly available code to re-train their model our dataset and extract body joint features of size 2 256 x 68 before the final classification layer (corresponding to two person instances). We apply a linear layer to transform these to 3 2 x 1024 input embeddings for pose BERT where 3 corresponds to the number of visual frames and 2 to maximum number of persons per frame.
graph by: (1) connecting human body joints within a
Audio Representations. Consistent with prior works, we use time- freeffreyeecyy representation of the input audio. We apply STFT [88] to generate the corresponding spectrogram and then transform the magnitudes of the spectrogram into the log- frequency scale for further processing. The size of the final input audio spectrogram is 256 256 and used in two ways: (1) as an audio embedding for audio BERT; and (2) as the input audio for attention U-ner for the task of sound source separation, which predicts individual audio spectrogram masks (see Figure D]. Before passing to audio BERT, we use a VGGish Network (22 to extract global features for input audio embedding.
http:s//ggihuub.ccc/ysssj//////
5

---

Tri-modal Co-attention. Recent works B 39 propose co-attentional transformer layers to generate effective representations of vision conditioned on language and vice versa.
In this paper, we introduce a tri-modal co- attentional layer, illustrated the right, by ex- lending ViLBERT's co-attentional transformer layers [33. Given intermediate representations for vision, pose, and audio, denoted as Hy (i), Hp(j), and (k()), respectively, each stream computes individual query (Q), key (K), and value (V) matrices. The keys and values from two modalities are then concatenated together and fed as input to the multi-head attention block of the third modality. As result, the block gen- erates attention features conditioned on the other two modalities. We keep the rest of the architec- ture, such as the feed forward layers, residual connections, etc. the same as a standard transformer block, which is then used to generate effective multi-modal features.
Huert
A Vare
Neme
Peon Farver
AcE Now
Mull seat
Millinas
Mail bood
Yeu
11:00
3.2.1 Training Tasks
We pre-train TriBERT jointly on two tasks: instrument classification and sound source separation. Our proposed architecture has three separate streams and each stream performs an individual classification task. To train our TriBERT model, we use the MUSIC21 dataset [56), which contains 21 instruments.
Weakly-supervised Visual and Pose Classification. Our visual segmentation network generates attention features for input video frames. We then apply a spatial pooling, and the resulting feature vector is fed into the visual BERT. We use a special <SOS> token at the beginning of the input frame sequence to represent the entire visual input. Following [33,, we apply masking to approximately 15% of the input image regions (see Figure The output of the visual BERT is sequence of hidden representations hvo, ..ww conditioned on the pose and audio modalities. We use mean pooling of all hidden representations to perform classification for the detected objects. Similarly, pose BERT generates a sequence of hidden representations hpp. .hhp conditioned on the visual and audio modalities, and we apply classification based on the mean pooling of all hidden states. Due to the lack of instance annotations, we cannot use region/pose level supervision. Following 5 we use a weakly- approach to perform region selection and classification.
Audio Classification. Since we do not have a sequence of audio embeddings, we artificially create an audio sequence for computational convenience by repeating the VGGish audio feature to generate a sequence of hidden representations hao, hal,.-hax conditioned on the visual and pose modalities. This is done purely for engineering convenience allow consistent use of tri-modal co-attention across modalities. We then apply audio classification on the mean feature of all hidden representations.
Multi-modal Sound Source Separation. We consider sound source separation as one of our initial tasks and follow the "Mix-and-Sepee and-Separrtte framework [T] 16 38 55), a well-known approach to solve this problem. The goal is to mix multiple audio signals to generate an artificially complex auditory representation and then learn to separate individual sounds from the mixture.
Given two input videos Vi and V/2 with accompanying audio A (!) and A2((), we mix A and A2 to generate a complex audio signal mixture (!) = A1 + A2(1). Suppose Vi has two objects on' and 01" " with accompanying audio 01 and a_ while V/2 has one object og' with audio az' The goal to separate sounds aj', a1", and a2' from the mixture Am (t) by predicting spectrogram masks using attention U-net (37), which takes in the mixed spectrogram as input. Attention U-net contains 7 convolutions and 7 de-convolutions with skip connections. The skip connections use attention gates (AG) comprise simple additive soft attentions to highlight relevant regions of the audio spectrograms. The overhead of attention U-Net over U-Net is fairly minimal. Specifically, in terms of the number of parameters, attention U-Net contains a modest 9% more parameters as compared to U-Net and the inference speed only 7% slower (37] The attention U-net outputs the final magnitude of the spectrogram mask (bottom branch in Figure T guided by audio-visual-Pe
features. Following [15] we adopt self-attention based early fusion between the bottle-neck of attention U-net with the fused features (i.e. concatenation of features) corresponding to the <SOS> tokens of three BERT streams.

---

We combine the predicted magnitude of the spectrogram mask from attention U-net with the phase of the input spectrogram and then use inverse STFT 18 to get back the wave-form the prediction.
Training Objective. We consider weakly-suupervised classification for the visual and pose modalities. two data streams from the hidden state of each modality. 'he first stream corresponds to a class score (3.lsss) for each individual region to perform recognition. This is achieved by a linear layer followed by a softmax operation (see Eq. 3 The second stream computes a probability distribution (Baer) for performing a proxy detection. This is done by using another linear layer followed by another softmax operation (see Eq as follows:
Following (5),
Pelass (h")is
(3)
Bao (h"):;
(4)
Σi ci
where h" E RCx1 ha E RCx|n and C' denotes the number of classes. We then aggregate the recognition and detection scores to predict the class of all image regions as follows: 3 Belaso(h" (h") @ Bder(ht (h"), where @ denotes an element-win wise product of the two scoring metrics. Finally we apply BCE-loss [10 to train visual and pose BERT. For audio classification, we consider classification layer predict audio classes and similarly apply BCE-loss to train audio BERT.
For the sound separation task, our goal is learn separate spectrogram masks for each individual object. Following [55] we use a binary mask which effectively corresponds to hard attention and use per-pixel sigmoid cross entropy loss (BCE-loss) to train the network.
Implementation Details. We used PyTorch to implement our network We consider thre random consecutive frames with size 224 x 224 x 3 our input sequence for visual and pose BERT and pre-trained ResNet50 (00 to extract global visual features for further processing. For the pose stream, first predict 2D coordinates of body and finger key points of each frame using AlphaPose [2] and then use graph CNN [52] to generate feature vectors for each keypoint. Similar to prior works [5] [5],
we sub-sample audio signals to 11KHz to reduce the computational cost and then select approximately 6s of audio by random cropping. To follow the Miix-and-Separaa mix audio inputs and generate a time-frequency audio spectrogram using STFT with Hann window size of 1022 and a hop length of 256. We then transform the spectrogram into the log- frequency scale to obtain the final 256 256 tmeefrequency representation. The transformers for visual/pose and audio have a hidden state size of 1024 and 512, respectively, with 8 attention heads. We use the Adam optimizer with an initial learning rate of -5 and batch size of to train the network on 4 GTX 1080 GPUs for 6k epochs. Training takes approximately 192 hours.
framework [1 17 38, (55),
3.2.2 Runtime Inference
We use the MUSIC21 I dataset [6 to train our network on two pretraining tasks: classification and sound source separation. We can this network directly for sound separation MUSIC21. We also fine-tune the pre-trained TriBERT on the MUSIC dataset (55) with 11 audio classes, which is sub-set of the MUSIC21 dataset. We follow fine-tuning strategy where we modify the classification layer from each pre-trained stream and then train our proposed model end-to-end with a learning rate of le for 1500 epochs while keeping the rest of the parameearsmeters the same as the initial task.
4 Experiments
Datasets. We consider the MUSIC21 dataset [56], which contains 1365 untrimmed videos of musical solos and duets from 21 instrument classes for the initial training of our TriBERT architecture. For fine-tuning, we use the MUSIC dataset (53), which is a subset of MUSIC21, containing 685 untrimmed videos of musical solos and duets from 11 instrument classes.
"htts://githubb. http://gggiuuu.co//hhiiiiiioooot com//uc-viisonnTTBBBE.
-BEsTd architectures, including ours, require large GPU memory and longer training time. Therefore, we use only three frames reduce computational cost, but the number of frames can be easily increased with the same architecture (if resources allow). Further, we would like to highlight that a pose feature for one frame, actually takes into account T=256 frames of poses using Spatial-Temporal Graph Convolutional Network. Therefore long- term contextual pose information is taken into account [52].
7

---

Table 1: Sound separation results on the MUSIC21 test SDR SIR / SAR are used report performance. Separation accuracy is captured by SDR and SIR; SAR only captures the absence of artifacts.
Single-Source
Multi-Source
Methods 5
SDR (f)
SIR (1)
SAR (1)
SDR (f)
SIR (f)
SAR (f)
Sound-of-Pixels
6.57
12.82
10.78
5.73
12.11
10.10
MUSIC- Gesture
15
8.08
15.27
11.29
6.72
14.03
9.68
Ours
10.09
17.45
12.80
7.66
14.54
11.06
Table 2: Sound separation results on the MUSIC test set. We use TriBERT with pre-trained
weights from the MUSIC21 dataset and then fine-tune this model using the MUSIC train set.
Methods NMF-MFCC 445
SDR (t)
SIR (1)
SAR (T)
0.92
5.68
6.84
AV-Mix-and- AV-Mix-and-Seeprrte [16
3.23
7.01
9.14
Sound-of-Pixell [55]
7.26
12.25
11.11
CO-SEPARATION
7.64
13.8
11.3
Mask Co-efficient with seg net 442
9.29
15.09
12.43
Ours (after fine-tune)
12.34
18.76
14.37
4.1 Experiments for Sound Separation
Evaluation Metries. We use three common metries quantify the performance of sound separation: Signal-to-Distortton Ratio (SDR), Signal-to-Interferenne Ratio (SIR), and Signal-to-Artfact Ratio (SAR). We report all of the results with the widely used mir_eval library
Baselines. The MUSIC21 I dataset contains 1365 untrimmed videos, but we found 314 of those to be missing. Moreover, the train/val/test split was unavailable. As a result, for fair comparison, trained our baselines [1555] 55 with the available videos using an 80/20 train/test split. We use publicly available code to train "Sound-of-Pa Pixells (55). For "MUSIC-Gesture" [155, we re-implemented the model by extracting pose features using Graph CNN [52]. Our reproduced results are comparable with those reported For the MUSIC dataset, follow the experimental protocol from [42] and consider their reported results as our baselines.
Quantitative and Qualitative Results. Table shows the quantitative results for the sound separa- tion pre- pre-training task on the MUSIC21 dataset. Here, we include the performance of our method and baselines when all models. Our TriBERT outperforms (10.09 vs 8.08 for single-source in SDR) baseline models in all evaluation metrics. We then fine-tune model on the MUSIC dataset with train/val/test split from [6] (see Table Our model again outperforms all baselines in all metrics (12.34 vs 9.29 in SDR). Figure 3 illustrates the corresponding qualitative results. The 1st, 2nd, and 3rd columns show the mixed video pairs and accompanying audio mixture, respectively. Columns 4 and 5 illustrate the ground-truth spectrogram mask while columns 6/7 and 8/9 show the predicted spectrogram mask by [15 and our method, respectively. Finally, the ground truth spectrogram, predicted spectrogram by [15] and method are illustrated in columns 10/11, 12/13, and 14/15, respectively. It is clear that TriBERT, both quantitatively and qualitatively, outperforms the state-of-the oftte----th-art in sound separation.
only single-source videos (solos) multi-source (solos+duets) to train
4.2 Multi-modal Retrieval
Retrieval Variants. In this experiment, we analyze the semantic alignment between the modalities that TriBERT learns to encode. This is done through cross-modal retrieval, where given a single or a pair of modality embeddings, we attempt to identify the matching embedding from a different modality. We consider 5 variants: audio - vision, vision - audio, audio - pose, pose - audio, and vision+audio > pose. Throughout this section, we refer to the embedding we have as the query
https:
"The reported SIR score in [3] is 15.81, which is close to our reimplementation of their method which achieves a score of 15.27. Our reproduced SDR score is bit lower. compared to the 10.12 reported in [15 However, this is perhaps expected given that 23% of the dataset missing.
8

---

Figure 3: Qualitative sound separation results on the MUSIC21 test set. Here, we show See text for details.
comparison between our results and Music-Gesture 15
embedding and the embedding we want to retrieve the result embedding. We train and evaluate on the MUSIC21 dataset, using the same 80-20 train-test split used to learn TriBERT.
We consider 2 types of embeddings for the modalities. First, use the transformer-basee embeddings, consisting of the concatenations of the hidden representations hwo....3. v3. hy....p3. and hao...a3 for visual, pose, and audio, respectively. Additionally, we establish a baseline by training with the embeddings used as input to the three BERT streams. This baseline can be viewed as an ablation study for the transformer layers.
Retrieval Training. Similar to [3], we train using an n-way multiple-chol choice setting. Here, n depends on the variant of the retrieval task, where n = 4 for the vision+audio to pose variant and n for the four remaining single-modality variants. In either case, one positive pair is used and 1 distractors are sampled. Further details are provided in the Supplemental Materials. We use an MLP that takes as input a fusion representation of both the query and result embeddings, computed as the dlement-wise product of the two. The module then outputs single logit, interpreted as a binary prediction for whether the query and result embeddings are aligned. For the vision+audio - pose variant, an additional MLP, based on [15], is used to combine the vision and audio embeddings before the final element- wise product with the pose embedding. Additionally, since both the transformer- based and pre- teeetranffommrr embeddings are not consistent in shape across the three modalities, use linear layers as required to transform them to a consistent one. This overall retrieval network is trained end-to-end. For each multiple choice, the network computes alignment score, after which a softmax is applied across all n scores. We train using a cross-entropy loss for 750 epochs with a batch size of 64 using the Adam optimizer with an initial learning rate of 2e-5.
also
Retrieval Results. Figure 4 shows the qualitative results for two variants of retrieval. Additionally, Table shows quantitative results for the 5 retrieval variants using the transformer-based representa- tion, the baseline pre-trame tre-transformer representation, and also a model that simply selects randomly from the pool. We see that retrieval using the transformer-bawee performance than the pre- treerrannfrrmr ones. This shows that the tri modal co-attention modules are an integral component in learning semantically meaningful relationship between the three modalities.
embeddings results in significantly better
Notably, in Table 3] we can see that vision+audio - pose worse than audio - pose in top-1 accuracy. The performance of the two models is not necessarily directly comparable. Specifically, there are two issues that should be considered:
The input dimensionality and number of parameters of the vision+audio retrieval model is signifi- cantly larger, with an additional MLP layer used for fusion. This means that the vision+audio model is more prone to over-fitting exhibited in the lower performance for top-1. Note that the top-5 and top-10 10 performance of vision+audio > pose is better.
The number of distractors (n 1) different in the two settings. For single-modality retrieval variants, we use two distractors (negative pairings); while for the two-modality variant, we use three distractors. This may also marginally affect the performance, since in the n-way classification, having more distractors puts more focus on the negatives.
However, we want to stress that the goal of these experiments is not to compare which modality or combination of modalities are best for retrieval. Instead, the goal is to illustrate the effectiveness of the TriBERT representations. Each of the five retrieval models is simply an instance a retrieval task. We can use any alternative (more sophisticated) models for retrieval here. The key observation is that in all five cases, TriBERT representations perform significantly better in retrieval compared with baseline representations (used input to TriBERT). This is strong evidence that TriBERT representations

---

Table 3: Multi-modal retrieval on the MUSIC21 test set. Top-k accuracy results for 5 retrieval variants, For each variant, retrieval on both the transformer-basee (bolded) and tre-transformer (unbolded) embeddings were evaluated. Also shown the accuracy of a random selection model.
Top-1 Accuracy (random 0.48)
Top-5 Accuracy
Top- Accuracy (random = 4.76)
Retrieval Variant
(random 2.38)
Audio + Vision
68.10
1,43
93.81
10.48
98.57
17.62
Vision + Audio
59.52
4.76
89.52
18.57
92.38
31.90
Audio Pose
63.81
2.38
86.67
6.67
94.29
10.95
Pose - Audio
54.29
1.90
85.24
9.05
86.67
14.29
Vision + Audio + Pose
54.29
3.33
90,00
9.05
96.19
15,24
We make no claims with regards to optimality of the retrieval formulation or objective; it is simply used as a proxy for evaluating TriBERT representations.
effective.
Query
Ground Truth
Top Result
Top 2 Result
Top 3 Result
Top Result
Top Result
to Vision
t0 Pose
Figure 4: Qualitative cross-modal retrieval results the MUSIC21 test set. The similarities (same instrument class) between results in the top-5 retrieval pool show that our transformer-he based representations have learned semantically meaningful relationship between the modalities.
Conclusion
In this paper, we introduce TriBERT, a three-stream model with tri-modal co-attention blocks to generate generic representation for multiple audio-visual tasks. We pre-train our model on the MUSIC21 dataset and show that our model exceeds state-of-the-an for sound separation We also find that TriBERT learns more generic and aligned cross-modal audio audio-visual -pose retrieval task. In this work, we limit ourselves to two datasets and fundamental audio-visusl tasks. In the future, we plan consider using more datasets and expanding to broader set of tasks (e.g., generation). The role of positional embeddings should also be explored.
representations, exceeding on the
Acknowledgments: This work was funded in part by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC Canada Research Chair (CRC) and an NSERC Discovery and Discovery Accelerator Supplement Grants.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute www vectorinstitute wwtpettittttttttitteeeteeeee wwtaaaaattaaerrree Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award.
References
[1] Relja Arandjelovic and Andrew Zisserman. Look, listen and leam. In 20/7 IEEE International Conference Computer Vision (ICCV), pages 609-617, 2017.
[2] Relja Arandjelović and Andrew Zisserman. Objects that sound, 2017.
[3] Björn Bebensee Byoung- Tak Zhang, Co-attentional transformers for story-based video understand- ing. In ICASSP 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4005-4009. IEEF, 2021.
-ak
'Sound source separation, as a task, can potentially have unintended privacy implications. For example, a conversation in crowded scene may no longer be considered private if approaches work well and deployed at large. We feel these concerns are benign at the moment, but should be revisited as technology evolves.
10

---

[4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828, 2013.
[5] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2846-28555 2016.
[6] Wim Boes and Hugo Van hamme. Audiovisual transformer architectures for large-scale classification and synchronization of weakly labeled audio events. In Proceedingy of the International Conference on Multimedia (ACM MM), 2019.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Eurupean Conference on Computer Vision, pages 213-229, 2020.
[8] I.. Chen, R. Maddox, Duan, and Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[9] Trevor Darrell, John W. Fisher, Paul A. Viola, and William T. Freeman. Audio-visual segmentation and "the cocktail party effect". In Proceedings of the Third International Conference Advances in Multimodal Interfaces, ICMI '00, page 32 40. Springer-Verteee 2000.
[10] Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross- entropy method. Annals of operations research, 134(1):19-67, 2005.
[11] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A seakerr-ndeeendent audio-vias model for speech separation. arXiv preprint
(eee:0044003999 2018
[12] Hao-Shu Fang. Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2334-2343, 2017.
[13] John W Fisher III, Trevor Darrell, William Freeman, and Paul Viola. Learning joint statistical models for audio-visual fusion and segregation. Information Processing Systems, volume 13. MIT Press, 2001.
Leen, T. Dietterich, and V. Tresp. editors, Advances Neural
[14] Aviv Gabbay, Ariel Ephrat, Tavi Halperin, and Shmuel Peleg. Seeing through noise: Visually driven speaker separation and enhancement. pages 3051-3055 04 2018.
[15] C. Gan. D. Huang, Zhao, J.B. Tenenbaum, and Torralba. Music gesture for visual sound separation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2020.
[16] R. Gao and K. Grauman. Co-separating sounds visual objects. In IEFF/CVF International Conference on Computer Vision (ICCV), pages 3879-3888, 2019.
[17] Ruohan Gao and Kristen Grauman. 2.5 visual sound. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324-333, 2019.
[18] Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transac- tions on acoustics, speech, and signal processing, 32(2):236-243 1984.
[19] Simon Haykin Zhe Chen. The cocktail party problem. Neural computation, 17:1875-902, 2005.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] John Hershey, H. Attias, Nebojsa Jojic, and Trausti Kristjansson. Audio-visual graphical models for speech processing. volume 5. pages V 649, 06 2004.
[22] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Scybold, et al. Can architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 13 131-135. IEEE, 2017.
[23] S. Hong, W. Im, and H. S. Yang. Cbvmr: Content-based vidco-music retrieval using soft intra-modal structure constraint. In ACM International Conference on Multimedia Retrieval (ICMR), 2018
[24] Sungeun Hong, Woobin Im, and Hyun S. Yang. Content-baan based video-music retrieval using soft intra-modal structure constraint. 2017.

---

[25] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[26] Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John Hershey. Single-channel multi- speaker separation using deep clustering. pages 545-549, 09 2016.
[27] Bruno Korbar, Du Tran, and Lorenzo Torresami. Cooperative learning of audio and video models from self- supervised synchronization. Proceedings of the 32nd International Conference on Neural Information Processing Systems, page 7774-7785, Red Hook, NY, USA, 2018. Curran Associates Inc.
[28] Sangho Lee, Youngjae Yu. Gunhee Kim, Thomas Breuel, and Jan Kautz. Parameter efficient multimodal transformers for video representation learning. In International Conference on Learning Representations (ICIR), 2021.
[29] Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter efficient multimodal transformers for video representation learning, 2020.
[30] Bochen Li. Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. See and listen: Score-informed association of sound tracks to players in chamber music performance videos. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2906-2910, 2017.
[31] G. Li, N. Duan, Y. Fang, M. Gong, D. Jiang, and M. Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
[32] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[33] Lu. J. and Batra, D. and Parikh, D. and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic rep- resentations for vision-and-languaae tasks. In Neural Information and Processing Systems (NeurIPS), 2019.
[34] Luo, Zhuo Chen, and Nima Mesgarani. Speaker-independeer speech separation with deep attractor network. IEEE/ACM Transactions on Audio, Speech, and Language Prucessing, 26(4):787-796, 2018.
[35] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio visual vidco representations. In International Conference on Learning Representations (ICIR), 2021.
[36] A. Nagrani, S. Albanie, and A. Zisserman. Learnable pins: Cross-modal embeddings for person identity.
In European Conference Computer Vision 2018.
[37] Ozan Oktay, Jo Schlemper, Loic Le Folgoc. Matthew Lee. Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, al. Attention u-net: Learning where look for the pancreas. arXiv preprint arXiv: (004:00049999 2018
[38] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with sule-supe multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 631-688, 2018
[39] S. Parekh, S. Essid, A. Ozerov, N. Q. K. Duong, P. Pérez, and G. Richar. Weakly supervised representation learning for audio-visual scene analysis. IEEE/ACM Transactions on Audio, Speech, and Language P Processing, 28, 2019.
[40] Jie Pu, Yannis Panagakis, Stavros Petridis, and Maja Pantic. Audio-visual object localization and separation using low-rank and sparsity. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2901-2905, 2017.
[41] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and Colin Raffel. mir eval: A transparent implementation of common mir metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR. Citeseer, 2014.
[42] T. Rahman and L.. Sigal. Suakly-supervised audio-visual sound source detection and separation. arXiv preprint arXiv: 104.0260, 2021.
[43] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba. Self-supervised audio-visual co-segmentation. pages 2357-2361, 05 2019.
[44] E. Shlizerman, L. Dery. H. Schoen, and I. Gernelmacher-Shiizzmman Audio to body dynamics. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018
12

---

[45] M. Spiertz and V. Gnann. Source-filter based clustering for monaural blind source separation. In International Conference Digital Audio Effects, 2009.
[46] W. Su, X. Zhu, Y. Cao, Li, I. Lu, F. Wei, and J. Dai. VI-bert: Pre-training of generic visual-linguistic representations. In International Conference Learning Representations (ICLR), 2020.
[47] Didac Surís, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i Nieto. Cross-modal embeddingg for video and audio retrieval. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0- 0, 2018.
[48] Didac Surís, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró Nieto. Cross-modal embeddings for video and audio retrieval. 2018.
[49] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint rrXiv::706.03762, 2017.
[51] Jianren Wang. Zhaoyuan Fang, and Hang Zhao. Alignnet: unifying approach to audio-visual alignment. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3309-3317,
2020.
[52] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[53] Donghuo Zeng, Yi Yu, and Keizo Oyama. Deep triplet neural networks with cluster-cea for audio-visue cross-modal retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(3):1 23, 2020.
[54] Tianyi Zhang, Guosheng Lin, Jianfei Cai. Tong Shen, Chunhua Shen, and Alex C Kot. Decoupled spatial neural attention for weakly supervised semantic segmentation. IEEE Transactions on Multimedia. 21(11) ((11):2930-299 2019.
[55] Zhao, C. Gan, A. Rouditchenko. C. Vondrick, McDermott, Torralba. The sound pixels. In European Conference on Computer Vision (ECCV). pages 570-580 586, 2018.
[56] Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba. The sound of motions. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[57] H. Zhu, M.o, R. Wang. A.H.. Zheng, and He. Deep audio-visual learning: A survey, International Journal of Automation and Computing, 18(3):351-376 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu-
tions and scope? [Yes] See abstract and contributions in Introduction (section
(b) Did you describe the limitations your work? [Yes] conclusion (section 5).
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See conclusion
(section 5)
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A] Our work theoretical work.
(b) Did you include complete proof's of theoretical results? [N/A] Our work application based. Therefore, we have included experiments in this paper.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or as URL)? [Yes] We will make our code and pretrained models publicly available upon acceptance of the paper.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters. how they were chosen)? [Yes]] For data splits and hyperparameters, and basclines (section 4.T]. respectively.
implementation details (section 3.2.1
(c) Did you report error bars (e.g., with respect the random seed after rumning experiments multiple times)? [No] We know that transformer based models are computationally expensive and take long time to train the whole network. Our model also takes approximately 192 hrs on GTX 1080 GPUs to train. Therefore, it is not feasible to train the model multiple times.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster. or cloud provider)? [Yes] See implementation details (section 3.2. Moreover,
we did not any cloud provider.
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) your work existing assets, did you cite the creators? [Yes]] See datasets in section 4
(b) Did you mention the license of the assets? [N/A] The assets does not provide any license.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [N/A] The asset (i.e. datasets) provides YoutTube video names and we did not violet YouTuhe license.
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "mNYsyXJDQwR", "reviewer": "Reviewer_WFQN", "summary": "By pointing out the recent transformer-based models are mostly designed for visual-language data, this paper introduce TriBERT which specifically targeting on audio-visual modalities to address such limitation. Inspired by ViLBERT, TriBERT is a transformer based model which enables contextual feature learning across three modalities: vision, pose and audio with the use of flexible co-attention. Besides, this paper introduce a learned visual tokenization scheme based on spatial attention and leverage weak-supervision to allow granular cross-modal interactions for visual and pose modalities. To demonstrate the proposed approach, TriBERT is pretrained on MUSIC221 and show improved performance in audio-visual sound source separation and other audio-visual-pose retrieval. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces TriBERT, a three-stream model which can accept three modalities, i.e. audio, visual, pose. By utilizing tri-modal co-attention blocks, generic audio-visual representations are achieved for audio-visual tasks of sound source separation and audio-visual-pose retrieval. This work of extending the usage of transformer-based models for more modalities other than just visual and language, i.e. audio-visual, is well motivated. \n\nHowever, my main concern is the novelty of this work is limited. Though, to extending transformer to audio and pose is not a trivial problem, the approach been used is straightforward. The whole TriBERT model is kind of a combination of some existing building blocks, e.g. BERT, graph CNN, segmentation net, Vggish net, UNet, etc. The training objective is designed as a multi-task manner, i.e. visual and pose classification, audio classification, and multi-modal sound source separation. From both aspects of model design, and algorithm/training objective, I do see much novelty appears. \nBesides, the experiments are not strong enough. First, with comparing with only a few SOTA approaches,  I can not be fully convinced. Also, why there no comparisons with sota for retrieval task? \nFor many recent audio-visual representation learning work, the approaches are usually demonstrated on 2-3 audio-visual tasks, e.g. audio event classification, and activity classification. I am wondering would these tasks can be considered to demonstrate the authors' claim of 'learning generic audio-visual representation'?\nI also did not see any ablation studies of the proposed approach. For example, the authors claim they proposed a new learned visual tokenization scheme, but no experimental results are shown to demonstrate the effectiveness of this tokenization scheme.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some of my suggestions:\n1) add more SOTA approaches for comparison on sound source localization task\n2) add SOTA comparisons on retrieval task\n3) add ablation studies of the proposed approach, e.g. comparisons of different visual tokenization approaches\n4) further demonstrate the claim of 'learning generic audio-visual representations', e.g. consider tasks of audio/video classification\n5) comparing with other audio-visual work, pose is the an extra modalities and the unique part in this paper. I would suggest to add some experiments and analysis on the contribution of adding pose\n6) no code was submitted. I am wondering the reproducibility of this work\n7) minor: alter positions of  Table 1 & 2 ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hrs"}, "extra_text": {}}, {"review_id": "dULdB1slNb1", "reviewer": "Reviewer_8Z91", "summary": "The goal of this work is to learn human-centric audio-visual representations. The authors claim to be first in using transformer models in the context of granular audio-visual detection or segmentation tasks (e.g. sound source separation). The authors propose TriBERT, which is a three-stream model with tri-modal attention to generate generic representaitons for AV tasks. The three modalities are audio, video and pose (keypoints). The model is trained with weak supervision (video level labels). The model gives state-of-the-art results on sound source separation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces BERT-like contextualized representation learning for the audio-visual domain. This is first in the context of AV source separation to my knowledge. The method appears to be effective for the task. \n\nThe architecture is based on existing work (VilBERT), but adds additional components such as segmentation network, spectrogram prediction mask and pose input.\n\nThe results are strong on the datasets shown. However, as the authors noted, the experiments are limited to only two datasets, one of which cannot be reproduced completely due to missing files (line 307). Ablations are also not given to justify design choices.\n\nThe results would be more convincing if the results could be reproduced for other human-centric datasets such as action recognition (UCF101) or audio-visual speech recognition (LRS3). \n\nDue to the limitations in experimental validation, I recommend borderline reject.\n\nThe English is fine, but the paper is not easy to follow (perhaps due to the large number of components to explain).\n\n\nAdditional questions:\n\nDoes the authors have an explanation as to why V+A>P is so much worse than A>P in top1 accuracy in Table 3?\n\nIsn't the pose just a processed form of the visual input? Is it just to make it more explicit so that it is easier for the network to process?\n\nWhat information is fed into the network in the case of occlusion?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Addressed in Section 5", "review_text": "", "extra_scores": {"ethical_concerns": "No concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2hr"}, "extra_text": {}}, {"review_id": "RKd9ZxrTVJM", "reviewer": "Reviewer_voNw", "summary": "The paper presents TriBERT a BERT-like model for learning multi-model representations of three modalities simultaneously, namely: vision, pose and audio. The architecture of TriBERT takes inspiration from the visual-linguistic model VilBERT (notably known for the applications in VQA) in terms of its Transformer architecture and of its multi-modal attention (the latter is extended in the paper to 3 modalities instead of 2).\n\nTriBERT training comprises four losses to optimize. In particular, after the projection of the 3 unimodal feature sequences into the joint feature space by the multimodal transformer, there are 3 sequences of features, namely: <h_v>, <h_p> and <h_a> for respectively enriched visual, pose and audio features. Then, the <h_v> and <h_p> features are (independently) trained to localize and predict the sound source (the musical instrument(s) which is (are) played in the recording) using 2 (independent) BCE losses. It should be noted that the localization part is learnt in a weakly-supervised manner as only global instrument classification annotations are available in the training data. The <h_a> audio features are trained to classify the sound source (the 3rd BCE loss).\n\nFinally, the 4th supervision task is the multi-modal sound source separation one. The original audio spectrogram is mixed with other audio spectrogram(s) and the task consists in predicting a spectrogram mask which would separate the original audio from the others (the 4th BCE loss). Following [37], for this last task, a U-Net neural architecture is used. U-Net takes the mixed audio spectrogram at its input and the fusion of <h_v>, <h_p> and <h_a> at its bottleneck representation to predict the desired spectrogram mask.\n\nOnce trained, TriBERT is evaluated on the MUSIC21 dataset for the sound separation task, and it also fine-tuned and evaluated for the multi-modal retrieval downstream task.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*\n\nThe paper has a primary original contribution: in particular, to the best of my knowledge, it is the 1st study to report a conclusive approach for training of a tri-modal BERT-like model on visual, audio and pose modalities with impressive results on such a complicate downstream task as multi-modal retrieval.\n\nThe paper has also two other minor contributions. Firstly, it demonstrates that unimodal visual features can be learnt from scratch in an end-2-end manner (while a common approach is to extract object features with a pretrained object detector such as Faster-RCNN as it is done in VilBERT, for example), and secondly, they illustrate that sound source prediction, separation and localization are powerful supervision tasks which are enough to train a complex Transformer architecture with relatively little annotations (only the tags of played instruments).\n\n*Quality*\n\nI believe that the authors have properly cited and compared their approach to all significant related work, most notable to [15] and [55]. Moreover, the approach of [15] has been re-implemented and retrained in order to use the same unimodal features for the pose modality (the ones learnt with a graph CNN on keypoints extracted with the AlphaPose toolbox).\n\nAll in all, in my opinion, the paper is technically sound.\n\n*Clarity*\n\nThe paper is very well written and easy to follow. The authors commit to publishing the source code upon the paper acceptance.\n\n*Typos*\n\n1.\tLine 93: “of of” - > “of”.\n\n*Significance*\n\nTo my knowledge, this is the 1st paper which has successfully managed to learn joint representations of vision, pose and audio inputs with a single BERT-like architecture. Moreover, only very shallow annotations (the musical instrument type) of the training dataset (MUSIC21) have been used for training. Given the convincing experimental results on the sound separation task where the strong Sound-of-Pixels [55] and MUSIC-Gesture [15] baselines have been largely outperformed, as well as the multi-modal retrieval results reported in Table 3 which demonstrate that TriBERT managed to learn a joint feature space for the three modalities, I believe that this paper will be useful for the community as a 1st example of what might become a new standard in audio-visual processing (as it was the case for VilBERT in the visual-linguistic domain). Therefore, I am rather inclined to recommend the acceptance of this paper.\n\nAt the same time, I would like to highlight the following limitations and suggestions / questions for the authors:\n\n1.\tThe experiments are done only on a single dataset (MUSIC is actually a subset of MUSIC21). The results would have been even more convincing, had there been another dataset for evaluation.\n\n2.\tI would be interested to see an ablation on the contributions of each modality to the final sound separation results reported in Tables 1 and 2. Thus, I believe that the pose and acoustic modalities are much more important than the visual one for this task. Does the visual modality really contribute in the final score?\n\n3.\tIt would also be informative to see some qualitative results on the sound localization given the fact that this task has been learnt in a weakly supervised manner (without explicit annotations on the localization).\n\n4.\tFinally, how can you explain that the retrieval Top-1 accuracy for the pose modality given only audio modality is higher than given both audio and visual modalities (Table 3)?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any negative social impact for this research.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "FCL_GOumbDS", "reviewer": "Reviewer_YXFS", "summary": "This work proposes TriBERT to handle the modalities of RGB image, poses and audios for tackling the task of sound source separation. The BERT model is used to extract features following ViLBERT. Experiments show appealing results on sound separation metrics compared with the author-reproduced previous SOTA methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper illustrates a relatively reasonable story in its introduction, however, the story cannot be fully supported by the experiments conducted. The usage of the BERT model in the field of visual sound separation is quite interesting. The model descriptions are mostly clear enough for reproducibility. The reviewer thinks this paper is of certain value to the field of visual sound separation.\n\n** The strengths of this paper are summarized as follows:\n\n++ This paper is well written and easy to follow.\n\n++ The idea of involving BERT in visual sound separation is interesting.\n\n++ Most of the model design is reasonable and with care, which is important for application papers.\n\n** The weaknesses of the paper are as follows:\n\n1. The conducted experiments do not match with the story in the introduction and the title.\n\na) The title is “Human-centric” audio-visual representation learning, however, the experiments are conducted on MUSIC21 dataset with sound source separation task. I am not saying that separating instrument sound is not entirely “human-centric”, but this is only a relatively small area in the  “human-centric” audio-visual field. I am expecting to see experiments conducted on human speech and its relationship with maybe co-speech gestures when I read the introduction. The experiments cannot support the title or the introduction at all. \n\nb) The core of ViLBERT or related work, is to use proxy tasks for training and show the effectiveness of learned features with downstream applications, while this paper basically focuses on sound separation. Although this is easy to understand: there are not so many downstream tasks if the model is trained on MUSIC dataset.\n\nThe reviewer suggests that the authors either focus on visual sound separation (with a title like “TriBERT for visual sound separation”) or conduct experiments on large-scale “human-centric” datasets such as TED talks or AudioSet, and show the effectiveness of representation learning on more downstream tasks.\n\n2. No qualitative results and no video supplementary shown. Demo videos are extremely important for audio-visual papers. With no video results, it is impossible for reviewers to know the true performance of the method. \n\nAlso, the authors do not provide any visual results of visual segmentation while the visual branch is designed for this. \nThe retrieval experiments are weird. Though I understand the authors are trying to match the “representation learning” in the title, these kinds of experiments are useless and have never been conducted in related studies.\n\n3. The quantitative experiments are not persuasive. Although our community cares less about computational cost, the paper lacks a lot of ablation studies. For example, this paper leverages attention U-Net as backbone, which is rarely used in previous works. The entire visual network is more complicated than all previous work. And the three modalities in the TriBERT are not evaluated efficiently. What if one of the modality is erased? More needed ablations are not listed.\n\n4. Details. Why use only three frames which is the same as PixelPlayer for the pose branch? This is not reasonable given that the pose frames should cover more temporal information as performed in Music Gesture.\n\nWhy Table1 is below Table2 and why evaluate on MUSIC alone given that it is a subset of MUSIC21? The settings are not reasonable. \n\n-------------------------------------\n\nThe rebuttal addresses some of my concerns and provides positive revision directions. However, it requires a major revision from the current form (the story needs to change, or large amounts of experiments should be shown), thus I still do not recommend acceptance of this paper. I raise my rating from 4 to 5 and encourage the authors to keep perfecting this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed their limitations. But these are major weaknesses of their work that could not be simply regarded as limitations. The authors should consider changing their story. \n\nVideo results should be included to show the qualitative results, so that more limitations could be revealed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new transformer-based model that can learn across three modalities: vision, pose, and audio for tackling the task of sound source separation. The authors did a good job during rebuttal and turned two slightly negative reviewers into positive ones.  The final score is very borderline (three borderline accepts and one borderline reject).  This work could clearly benefit from a more extensive experimental evaluation, but AC feels this work is very interesting and deserves to be published on NeurIPS. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are also encouraged to make other necessary changes.
Decision: Accept (Poster)