Title: Weakly-Supervised Audio-Visual Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: audio-visual learning

visual sound localization

audio-visual segmentation
Abstract: Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video.
  Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision.  However, these pixel-level masks are expensive and not available in all cases.
  In this work, we aim to simplify the supervision as the instance-level annotation, $\textit{i.e.}$, weakly-supervised audio-visual segmentation.
  We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation.
  Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.

Weakly-Supervised Audio-Visual Segmentation

Abstract

Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e. , weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on A VSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.

1 Introduction

When we hear a dog barking, we are naturally aware of where the dog is in the room due to the strong correspondence between audio signals and visual objects in the world. This human perception intelligence attracts many researchers to explore audio-visual joint learning for visual sound localization and segmentation. In this work, we aim to segment sound source masks from both frames and audio in a video without relying on expensive annotations of pixel-level masks, i.e. , only the instance-level annotation is given for training.

Audio-visual segmentation (AVS) [1] is a recently rising problem that predicts a pixel-level map of objects that produce sound at the time of frames. A related problem to this recent task is sound source localization, which aims to locate visual regions within the frame that correspond to the sound. Sound source localization (SSL) [2, 3, 4, 5, 6, 7, 8, 9, 10] estimates a rough location of the sounding objects at a patch level, while the goal of A VS is to estimate pixel-wise segmentation masks for all sound sources. AVS requires a more comprehensive manually designed architecture than SSL since it needs to identify the exact shape of sounding objects. However, this complicated network for AVS requires countless pixel-wise accurate masks as supervision, which costs numerous labor and resources. In contrast, we aim to get rid of these costly pixel-level annotations to achieve weakly-supervised audio-visual segmentation.

Many weakly-supervised methods [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] have been proposed to be successful in the visual segmentation community, where they replaced accurate pixel-level annotations with weak labels, including scribbles [13, 15], bounding boxes [11, 12], points [14], and image-level class labels [16, 17]. Typically, CAM [18] was proposed with a CNN-based image classifier to generate pseudo segmentation masks at the pixel level using coarse object localization maps. With coarse localization maps, following works tried to obtain the full extent of objects by regions expansion [21, 22, 23], stochastic inference [24], boundary constraints [25, 26, 27], or object regions mining and erasing [28, 29]. However, only one modality, i.e. , image is involved in these aforementioned CAM-based weakly-supervised approaches. In this work, we need to tackle with two distinctive modalities (audio and vision), where a well-designed cross-modal fusion is required to aggregate audio-visual features with high similarity in the semantic space.

The main challenge is that we do not have pixel-level segmentation masks for this new weaklysupervised multi-modal problem during training. Without this supervision, the performance of AVS [1] on audio-visual segmentation degrades significantly, as observed in our experiments in Section 4.2. In the meanwhile, previous weakly-supervised semantic segmentation [18, 30] and visual sound source localization [9] methods have two main challenges. First, no multi-modal constraint is applied for pixel-wise mask prediction in the previous weakly-supervised semantic segmentation methods as they do not use the audio as input during training. Second, current visual sound source localization baselines generate only coarse heatmaps for sounding objects instead of binary pixelwise segmentation masks. To address the aforementioned challenges, our key idea is to apply multi-scale multiple-instance contrastive learning in audio-visual fusion to learn multi-scale crossmodal alignment, which differs from existing weakly-supervised semantic segmentation approaches. During training, we aim to use a reliable pseudo mask with multi-scale visual features as mask-level supervision for predicting accurate audio-visual segmentation masks in a weakly-supervised setting.

To this end, we propose a novel weakly-supervised audio-visual segmentation framework, namely WS-VAS, that can predict sounding source masks from both audio and image without using the ground-truth masks. Specifically, our WS-AVS leverages multi-scale multiple-instance contrastive learning in audio-visual fusion to capture multi-scale cross-modal alignment for addressing modality uncertainty in weakly-supervised semantic segmentation methods. Then, the pseudo mask refined by contrastive class-agnostic maps will serve as pixel-level guidance during training. Compared to previous visual sound source localization approaches, our method can generate binary accurate audio-visual segmentation masks.

Empirical experiments on AVSBench [1] comprehensively demonstrate the state-of-the-art performance against previous weakly-supervised baselines. In addition, qualitative visualizations of segmentation masks vividly showcase the effectiveness of our WS-AVS in predicting sounding object masks from both audio and image. Extensive ablation studies also validate the importance of audiovisual fusion with multi-scale multiple-instance contrastive learning and pseudo mask refinement by contrastive class-agnostic maps in single-source and multi-source audio-visual segmentation.

Our main contributions can be summarized as follows:

We investigate a new weakly-supervised multi-modal problem that predicts sounding object masks from both audio and image without needing pixel-level annotations.

We present a novel framework for weakly-supervised audio-visual segmentation, namely WS-AVS, with multi-scale multiple-instance contrastive learning to capture multi-scale audio-visual alignment.

Extensive experiments comprehensively demonstrate the state-of-the-art superiority of our WS-AVS over previous baselines on single-source and multi-source sounding object segmentation.

2 Related Work

Audio-Visual Learning. Audio-visual learning has been explored in many existing works [31, 32, 33, 34, 2, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46] to learn audio-visual correspondence from those two distinct modalities in videos. Given video sequences, the goal is to push away those from different pairs while closing audio and visual representations from the same pair. Such cross-modal correspondence is useful for several tasks, such as audio/speech separation [37, 47, 48, 35, 36, 49, 50, 51, 52], visual sound source localization [2, 3, 4, 5, 6, 7, 8, 9, 10, 53, 54], audio spatialization [55, 56, 57, 38], and audio-visual parsing [58, 59, 60, 61]. In this work, we mainly focus on learning audio-visual association for pixel-level audio-visual segmentation, which is more demanding than those tasks mentioned above.

Visual Sound Localization . Visual sound source localization aims to identify objects or regions of a video that correspond to sounds in the video. Recent researchers [2, 4, 5, 7, 8, 62, 9, 10] used diverse networks to learn the audio-visual alignment for predicting sound sources in the video. Typically, Attention10k [2] proposed a two-stream architecture based on an attention mechanism to detect sounding objects in the image. More recently, EZVSL [9] applied a multiple-instance contrastive learning objective to learn the alignment across regions with the most corresponding audio. However, they cannot predict exact masks for sounding objects.

Audio-Visual Segmentation. Audio-visual segmentation is a challenging problem that predicts pixelwise masks for sounding objects in the image. In order to address this issue, Zhou et al. [1] introduced the first audio-visual segmentation benchmark with pixel-level annotations for each 5-second video, where a binary mask is used to indicate the pixels of sounding objects for the corresponding audio. Moreover, they proposed an encoder-decoder network with a temporal pixel-by-pixel audio-visual interaction module to solve the pixel-level audio-visual segmentation task. However, their approach requires exhausted pixel-level annotations as the main supervision. Without this supervision, their performance degrades significantly as observed in our experiments in Section 4.2. In this work, we aim to get rid of the dependency of pixel-level labels, by leveraging only instance-level labels for weakly-supervised audio-visual segmentation.

Weakly-Supervised Semantic Segmentation. Weakly-supervised visual segmentation aims at reducing the burden of collecting pixel-level annotations for image segmentation at a large scale required by its fully-supervised counterpart. Early methods explored different forms of weak supervision, including image-level label [63, 64], scribble [13, 15], and bounding box [12, 65, 66]. Typically, CAM [18] was introduced to generate class activation maps as pseudo pixel-level segmentation masks for weakly-supervised object localization. Based on CAM, following works tried to optimize the coarse class activation map for more precise object localization, by expanding localization regions [21, 22, 23], using stochastic inference [24], exploring boundary constraints [25, 26, 27], or mining and erasing regions of objects [28, 29]. More recently, C 2 AM [30] leveraged contrastive learning with the semantic relation between foreground and background as positive and negative pairs to generate a class-agnostic activation map. Different from weakly-supervised visual segmentation baselines, we aim to develop a fully novel framework to aggregate semantics from both audio and visual representations with the only instance-level annotation. To the best of our knowledge, we are the first to explore class-agnostic activation maps on weakly-supervised audio-visual segmentation. In addition, we leverage audio-visual multi-scale fusion to boost the segmentation performance.

Weakly-Supervised Audio Learning. Weakly-supervised learning has been widely-used in many audio-relevant tasks, such as sound events detection [67, 68, 69] and sound source separation [70, 71, 72, 73]. Typically, WAL-Net [69] proposed using weakly labeled data from the web to explore how the label density and corruption of labels affect the generalization of models. DC/GMM [71] applied an auxiliary network to generate the parameters of Gaussians in the embedding space with a one-hot vector indicating the class as input. However, the problem we need to solve in this work is more difficult and challenging. Indeed, we aim to leverage the weakly labelled supervision of image categories to solve the new weakly-supervised multi-modal problem.

3 Method

Given a clip of audio and video frames, our target is to predict pixel-level masks for sounding objects in the frame. We present a novel Weakly-Supervised Audio-Visual Segmentation framework named WS-AVS for single source audio-visual segmentation without involving the pixel-wise annotation, which mainly consists of two modules, Audio-Visual Fusion with Multi-scale Multiple Instance Contrastive Learning in Section 3.2 and Pseudo Mask Refinement by contrastive class-agnostic maps in Section 3.3.

3.1 Preliminaries

In this section, we first describe the problem setup and notations, and then revisit the first work [1] with pixel-level annotations for single source audio-visual segmentation.

Problem Setup and Notations. Given an audio spectrogram and an image, our goal is to predict the binary segmentation mask for the sound source in the image spatially. Let D = ( a i , v i ) : i = 1 , ..., N be a dataset of paired audio a i ∈ R T × F and visual data v i ∈ R 3 × H × W , where only one sound source a i are assumed to be existing in v i . Note that T, F denotes the dimension of time and frequency of the audio spectrogram. We follow previous work [1, 9] and first encode the audio and visual inputs using a two-stream neural network encoder and projection heads, denoted as f a ( · ) , g a ( · ) and f v ( · ) , g v ( · ) for the audio and images, separately. The audio encoder computes global audio representations a i = g a ( f a ( a i )) , a i ∈ R 1 × D and the visual encoder extracts multi-scale representations { v s i } S s =1 = g v ( f v ( v i )) , v s i ∈ R D × H s × W s for each s th stage. During the training, we do not have mask-level annotations Y ∈ R H × W .

Image

Audio

Pixel-wise

Loss

Audio

Encoder

Pseudo Mask

Refinement

multi-scale visual features

Pool

Image

Encoder

Audio-Visual

Fusion

Decoder

mask prediction

pseudo mask

Other

Image

Image

Encoder

Audio-Visual

Fusion

Multi-scale Multiple

Instance Contrastive Loss

positive

negative

global audio feature

!

!

"

"

!

!

#

"

#

$

%

&

!

"

Revisit Audio-Visual Segmentation with Pixel-level Annotations. To solve the single source audio-visual segmentation problem, AVS [1] introduced the pixel-wise audio-visual fusion module to encode the multi-scale visual features and global audio representations. After the cross-modal fusion, the audio-visual feature map z s i at the s th stage is updated as:

where ˆ a i ∈ R D × H s × W s denotes the duplicated version of the global audio representation a i that repeats H s W s times at the s th stage. µ ( · ) , θ ( · ) , ϕ ( · ) , ω ( · ) denote the 1 × 1 × 1 convolution operator. Then those updated multi-stage feature maps are passed into the decoder of Panoptic-FPN [74] and a sigmoid activation layer to generate the final output mask M ∈ R H × W . With the pixel-level annotation Y as supervision, they applied the binary cross entropy loss between the prediction and label as:

However, such a training mechanism is extremely dependent on pixel-level annotations. Without this supervision, their performance on single source audio-visual segmentation deteriorates significantly, as observed in Section 4.2. In the meanwhile, current weakly-supervised semantic segmentation (WSSS) [18, 30] and visual sound source localization (VSSL) [9] methods will pose two main challenges. First, there is no multi-modal constraint for pixel-level mask prediction in the previous WSSS methods as they do not involve the audio during training. Second, existing VSSL approaches only predict coarse heatmaps for sounding sources instead of accurate pixel-wise segmentation masks. To address these challenges, we propose a novel weakly-supervised audio-visual segmentation framework for single sounding object segmentation, which is composed of Audio-Visual Fusion with multi-scale multiple-instance contrastive learning and Pseudo Mask Refinement by contrastive class-agnostic map, as shown in Figure 1.

3.2 Audio-Visual Fusion with Multi-scale Multiple-Instance Contrastive Learning

In order to address the modality uncertainty brought by the previous WSSS baselines [18, 30], inspired by EZ-VSL [9], we propose to conduct audio-visual fusion by focusing only on the most

aligned regions when matching the audio to the multi-scale visual feature. This is because most locations in the video frame do not correspond to the sounding object, and thus the representations at these locations should not be aligned with the audio during training.

To align the audio and visual features at locations corresponding to sounding sources, we apply the multi-scale multiple-instance contrastive learning (M 2 ICL) objective to align at least one location in the corresponding bag of multi-scale visual features with the audio representation in the same mini-batch, which is defined as:

where the similarity sim ( a i , v s ) denotes the max-pooled audio-visual cosine similarity of a i and v s i across all spatial locations at s stage. That is, sim ( a i , v s i ) = max H s W s ( a i , v s i ) . B is the batch size, D is the dimension size, and τ is a temperature hyper-parameter.

Similar to EZ-VSL [9], we use a symmetric loss of Eq. 4 to discriminate negative audio bags from other audio samples in the same mini-batch, which is defined as

where a m denote the global audio visual from other sample m in the mini-batch. The overall audiovisual fusion objective with the multi-scale multiple-instance contrastive learning mechanism is defined as:

Optimizing the loss will push the model to learn discriminatively global audio representation a i and multi-scale visual features { v s i } S s =1 . Then, these features are fed forward into Eq. 1 to generate updated multi-scale audio-visual features { z s i } S s =1 . Finally, with these updated multi-scale audiovisual features, we apply the decoder of Panoptic-FPN [74] to generate the output mask M . Note that, the recent audio-visual segmentation baseline [1] does not give any cross-modal constraint on the audio and visual representations in the audio-visual fusion, which causes a significant performance drop when removing the ground-truth masks during training.

3.3 Pseudo Mask Refinement by Contrastive Class-agnostic Maps

The second challenge of spatial uncertainty in VSSL approaches [9] requires us to utilize mask-level supervision for generating precise final output mask M . To generate reliable pseudo masks of training sets, motivated by recent WSSL pipelines [30], we introduce the Pseudo Mask Refinement by applying a contrastive class-agnostic map on multi-scale visual features { v s i } S s =1 . Specifically, we utilize the contrastive loss in [30] to close the distance between the representations in positive pairs (foreground-foreground, background-background) and push away the representations in negative pairs (foreground-background). Then, we use the background activation maps as pseudo labels to further train a salient object detector [75] to predict the salient region S ∈ R 1 × H × W in the image. Finally, the predicted salient cues are concatenated with the initial class-agnostic map A ∈ R D × H × W to perform the argmax operator along the embedding dimension to generate the binary pseudo mask ˆ Y ∈ R H × W , which is formulated as:

where [ ; ] denotes the concatenation operator. Note that since we are focusing on single-source segmentation, L = [ L S ; L A ] ∈ R (1+ D ) × H × W . L S ∈ R 1 × H × W is the full label of zeros for the salient cues, while L A ∈ R D × H × W is the full label of ones for the class-agnostic maps. With the

reliable pseudo mask refinement, we apply the binary cross entropy loss between the predicted mask M and the pseudo mask ˆ Y , which is defined as:

The overall objective of our model is simply optimized in an end-to-end manner as:

During inference, we follow previous work [1] and directly use the predicted mask M ∈ R H × W as the final output. It is worth noting that the final localization map in the current VSSL methods [9] was generated through bilinear interpolation of the audio-visual feature map at the last stage.

4 Experiments

4.1 Experiment Setup

Datasets. AVSBench [1] contains 4,932 videos with 10,852 total frames from 23 categories including animals, humans, instruments, etc. Following prior work [1], we use the split of 3,452/740/740 videos for train/val/test in single source segmentation.

Evaluation Metrics. Following previous work [1], we use the averaged IoU (mIoU) and F-score to evaluate the audio-visual segmentation performance. mIoU computes the intersection-over-union (IoU) of the predicted mask and ground-truth mask for evaluating the region similarity. F-score calculates both the precision and recall for evaluating the contour accuracy.

Implementation. The input image is resized to a resolution of 224 × 224 . The input audio takes the log spectrograms extracted from 3s of audio at a sample rate of 22050Hz. Following previous works [9, 10], we apply STFT to generate an input tensor of size 257 × 300 (257 frequency bands over 300 timesteps) using 50ms windows with a hop size of 25ms. We follow the prior audio-visual segmentation work [1] and use the ResNet50 [76] as the audio and visual encoder. The visual model is initialized using weights pre-trained on ImageNet [77]. The model is trained with the Adam optimizer with default hyper-parameters β 1 = 0.9, β 2 = 0.999, and a learning rate of 1e-4. The model is trained for 20 epochs with a batch size of 64.

4.2 Comparison to Prior Work

In this work, we propose a novel and effective framework for weakly-supervised audio-visual segmentation. To validate the effectiveness of the proposed WS-AVS, we comprehensively compare it to previous audio-visual segmentation, weakly-supervised semantic segmentation, and visual sound source localization baselines: 1) A VS [1](ws): the weakly-supervised version of the first audio-visual segmentation work, where we removed the pixel-level annotations for training; 2) CAM [18]: the first baseline using class-activation maps as pseudo segmentation masks for weakly-supervised object localization; 3) EZ-VSL [9]: the state-of-the-art visual sound source localization approach with coarse source maps as output; 4) C 2 AM[30]: the very recent work for weakly-supervised semantic segmentation with only the image as input.

The quantitative comparison results are reported in Table 1. As can be seen, we achieve the best performance in terms of all metrics compared to previous weakly-supervised baselines. In particular,

Ground-Truth

Raw Image

AVS (ws)

EZ-VSL

CAM

C

!

AM

WS-AVS (ours)

the proposed WS-AVS significantly outperforms AVS [1](ws), the weakly-supervised version of the state-of-the-art audio-visual segmentation approach, by 21.50 mIoU and 26.77 F-score. Moreover, we achieve superior performance gains of 3.26 mIoU and 15.21 F-score compared to C 2 AM[30], which implies the importance of the proposed multi-scale multiple-instance contrastive learning for addressing the modality uncertainty in audio-visual fusion. Meanwhile, our WS-A VS outperforms EZVSL[9], the current state-of-the-art visual sound source localization baseline, by a large margin, where we achieve the performance gains of 4.73 mIoU and 16.06 F-score. These significant improvements demonstrate the superiority of our method in single-source audio-visual segmentation.

In addition, significant gains in multi-source sound localization can be observed in Table 1. Compared to AVS [1](ws), the weakly-supervised version of the state-of-the-art audio-visual segmentation approach, we achieve the results gains of 22.09 mIoU and 31.15 F1 score. Furthermore, the proposed approach still outperforms C 2 AM[30] by 5.52 mIoU and 17.29 F1 score. We also achieve highly better results than EZ-VSL [9], the current state-of-the-art visual sound source localization baseline. These results validate the effectiveness of our approach in learning multi-scale multiple-instance semantics from mixtures and images for multi-source localization.

In order to qualitatively evaluate the localization maps, we compare the proposed WS-AVS with weakly-supervised AVS [1], CAM [18], C 2 AM[30], and EZ-VSL [9] on single-source segmentation in Figure 2. From comparisons, three main observations can be derived: 1) Without pixel-level annotations, AVS [1], the first audio-visual segmentation fails to predict the masks for sounding objects; 2) the quality of segmentation masks generated by our method is much better than the strong visual sound source baseline, EZ-VSL [9]; 3) the proposed WS-AVS achieves competitive even better results on predicted masks against the weakly-supervised semantic segmentation baseline [30] by using contrast class-agnostic maps for prediction. These visualizations further showcase the superiority of our simple WS-AVS with multi-scale multiple-instance contrastive learning to guide segmentation for predicting accurate source masks.

4.3 Experimental Analysis

In this section, we performed ablation studies to demonstrate the benefit of introducing the AudioVisual Fusion with multi-scale multiple-instance contrastive learning and Pseudo Mask Refinement module by contrastive class-agnostic map. We also conducted extensive experiments to explore the effect of fusion stages and batch size on weakly-supervised audio-visual segmentation. Furthermore,

60

60

040

40

mloU

mloU

F-score

F-score

20

1

2

3

4

20

8

16

32

64

128

Fusion Stage (S)

we visualized high-quality pseudo masks from Pseudo Mask Refinement in weakly-supervised training.

Audio-Visual Fusion & Pseudo Mask Refinement. To validate the effectiveness of the introduced audio-visual fusion with multi-scale multiple-instance contrastive learning (A VF) and Pseudo Mask Refinement by contrastive class-agnostic map (PMR), we ablate the necessity of each module and report the quantitative results in Table 2. We can observe that adding AVF to the vanilla baseline highly increases the results of single-source audio-visual segmentation by 19.22 mIoU and 14.79 F-score, which demonstrates the benefit of multi-scale multiple-instance contrastive learning (M 2 ICL) in extracting aligned cross-modal feature in fusion for source segmentation. Meanwhile, introducing only PMR in the baseline also increases the segmentation performance in terms of all metrics. More importantly, incorporating A VF with M 2 ICL and PMR together into the baseline significantly raises the performance by 21.50 mIoU and 26.77 F-score. These improving results validate the importance of AVF with M 2 ICL and PMR with contrastive class-agnostic map in addressing both modality and spatial challenges for generating accurate audio-visua masks.

Effect of Fusion Stages and Batch Size. The number of fusion stages and batch size used in the proposed M 2 ICL affect the extracted cross-modal representations for audio-visual segmentation. To explore such effects more comprehensively, we varied the number of fusion stages from { 1 , 2 , 3 , 4 } and ablated the batch size from { 8 , 16 , 32 , 64 , 128 } . The comparison results of segmentation performance are shown in Figure 3. When the number of fusion stages is 4 and using batch size of 64 in M 2 ICL, we achieve the best segmentation performance in terms of all metrics. With the increase of fusion stages from 1 to 4, the proposed WS-AVS consistently raises results, which shows the importance of multi-scale visual features in audio-visual fusion for learning discriminative cross-modal representations. Regarding the batch size, the performance of the proposed WS-A VS climbs with the increase of the batch size from 8 to 64. However, increasing the batch size from 64 to 128 will not continually improve the result since there might be some false negatives in the mini-batch for this training set with a relatively smaller size.

Generated Pseudo Mask. Generating reliable pseudo masks with contrast class-agnostic maps is critical for us to train the weakly-supervised framework. To better evaluate the quality of generated pseudo masks, we visualize the pseudo mask and class-agnostic map in Figure 4. As can be observed in the last column, the class-agnostic map can successfully localize the sounding object in the image. Furthermore, the pseudo masks in the third column predicted by Eq. 6 have high quality and they are even fine-grained compared to the manually annotated ground-truth masks. These meaningful visualization results further showcase the success of the Pseudo Mask Refinement in extracting reliable pseudo masks as guidance for single-source audio-visual segmentation.

Ground-Truth

Raw Image

Pseudo Mask

Class-agnostic Map

Ground-Truth

Raw Image

Pseudo Mask

Class-agnostic Map

4.4 Limitation

Although the proposed WS-AVS achieves superior results on single-source audio-visual segmentation, the performance gains of our approach on the mIoU metric are not significant. One possible reason is that our model easily overfits across the training phase, and the solution is to incorporate dropout and momentum encoders together for weakly-supervised audio-visual segmentation. Meanwhile, we notice that our model performs worse on mixed sound sources, such as a scenario with both female singing and playing tabla. The future work could be to add separation objectives or assign semantic labels to both audio and visual segments in audible videos.

5 Conclusion

In this work, we successfully investigate the weakly-supervised audio-visual segmentation to eliminate pixel-level annotations as supervision. Then we present WS-AVS, a novel Weakly-Supervised Audio-Visual Segmentation framework with multi-scale multiple-instance contrastive learning for capturing multi-scale audio-visual alignment. Furthermore, we conduct extensive experiments on AVSBench to demonstrate the superiority of our WS-AVS against previous weakly-supervised audiovisual segmentation, weakly-supervised semantic segmentation, and visual sound source localization approaches.

Broader Impact. The proposed approach successfully predicts segmentation masks of sounding sources from manually-collected datasets on the web, which might cause the model to learn internal biases in the data. For instance, the model could fail to discover rare but crucial sound sources. Therefore, these issues should be addressed for the deployment of real applications.

References

Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision , 2022. 1, 2, 3, 4, 5, 6, 7

Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4358-4366, 2018. 1, 2

Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh H. McDermott, and Antonio Torralba. Self-supervised audio-visual co-segmentation. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2357-2361, 2019. 1, 2

Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9248-9257, 2019. 1, 2

Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Selfsupervised learning of audio-visual objects from video. In Proceedings of European Conference on Computer Vision (ECCV) , pages 208-224, 2020. 1, 2

Janani Ramaswamy and Sukhendu Das. See the sound, hear the pixels. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 2959-2968, 2020. 1, 2

Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Proceedings of European Conference on Computer Vision (ECCV) , pages 292-308, 2020. 1, 2

Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16867-16876, 2021. 1, 2

Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Proceedings of European Conference on Computer Vision (ECCV) , page 218-234, 2022. 1, 2, 3, 4, 5, 6, 7

Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2022. 1, 2, 6

George Papandreou, Liang-Chieh Chen, Kevin P Murphy, and Alan L Yuille. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In Proceedings of the IEEE international conference on computer vision , pages 1742-1750, 2015. 1

Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In Proceedings of the IEEE international conference on computer vision , pages 1635-1643, 2015. 1, 3

Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3159-3167, 2016. 1, 3

Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In European conference on computer vision , pages 549-565. Springer, 2016. 1

Paul Vernaza and Manmohan Chandraker. Learning random-walk label propagation for weaklysupervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7158-7166, 2017. 1, 3

Qibin Hou, Daniela Massiceti, Puneet Kumar Dokania, Yunchao Wei, Ming-Ming Cheng, and Philip HS Torr. Bottom-up top-down cues for weakly-supervised semantic segmentation. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition , pages 263-277. Springer, 2017. 1

Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1568-1576, 2017. 1

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921-2929, 2016. 1, 2, 3, 4, 6, 7

Tianfei Zhou, Wenguan Wang, Si Liu, Yi Yang, and Luc Van Gool. Differentiable multigranularity human representation learning for instance-aware human semantic parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1622-1631, 2021. 1

Jiapei Feng, Xinggang Wang, and Wenyu Liu. Deep graph cut network for weakly-supervised semantic segmentation. Science China Information Sciences , 64(3):1-12, 2021. 1

Alexander Kolesnikov and Christoph H Lampert. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In European conference on computer vision , pages 695-711. Springer, 2016. 1, 3

Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. Weaklysupervised semantic segmentation network with deep seeded region growing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7014-7023, 2018. 1, 3

Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang. Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7268-7277, 2018. 1, 3

Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5267-5276, 2019. 1, 3

Wataru Shimoda and Keiji Yanai. Self-supervised difference detection for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5208-5217, 2019. 1, 3

Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12275-12284, 2020. 1, 3

Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang. Weakly-supervised semantic segmentation via sub-category exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8991-9000, 2020. 1, 3

Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9215-9223, 2018. 1, 3

Qibin Hou, PengTao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-erasing network for integral object attention. Advances in Neural Information Processing Systems , 31, 2018. 1, 3

Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen. C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 989-998, 2022. 2, 3, 4, 5, 6, 7

Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2016. 2

Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 801-816, 2016. 2

Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 609-617, 2017. 2

Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2018. 2

Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 570-586, 2018. 2

Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1735-1744, 2019. 2

Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10478-10487, 2020. 2

Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , pages 4733-4744, 2020. 2

Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12934-12945, 2021. 2

Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12475-12486, June 2021. 2

Shentong Mo and Yapeng Tian. Semantic-aware multi-modal grouping for weakly-supervised audio-visual video parsing. In European Conference on Computer Vision (ECCV) Workshop , 2022. 2

Shentong Mo and Pedro Morgado. Benchmarking weakly-supervised audio-visual sound localization. In European Conference on Computer Vision (ECCV) Workshop , 2022. 2

Shentong Mo, Jing Shi, and Yapeng Tian. DiffAVA: Personalized text-to-audio generation with visual alignment. arXiv preprint arXiv:2305.12903 , 2023. 2

Shentong Mo and Pedro Morgado. A unified audio-visual learning framework for localization, separation, and recognition. arXiv preprint arXiv:2305.19458 , 2023. 2

Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. arXiv preprint arXiv:2308.11073 , 2023. 2

Shentong Mo, Weiguo Pian, and Yapeng Tian. Class-incremental grouping network for continual audio-visual learning. 2023. 2

Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 35-53, 2018. 2

Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3879-3888, 2019. 2

Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10457-10467, 2020. 2

Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2745-2754, 2021. 2

Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with crossmodal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15495-15505, 2021. 2

Tanzila Rahman and Leonid Sigal. Weakly-supervised audio-visual sound source detection and separation. In 2021 IEEE International Conference on Multimedia and Expo (ICME) , pages 1-6. IEEE, 2021. 2

Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. arXiv preprint arXiv:2303.17056 , 2023. 2

Shentong Mo and Yapeng Tian. AV-SAM: Segment anything model meets audio-visual localization and segmentation. arXiv preprint arXiv:2305.01836 , 2023. 2

Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360°video. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2018. 2

Ruohan Gao and Kristen Grauman. 2.5d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 324-333, 2019. 2

Changan Chen, Unnat Jain, Carl Schissler, S. V. A. Garí, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Proceedings of European Conference on Computer Vision (ECCV) , pages 17-36, 2020. 2

Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weaklysupervised audio-visual video parsing. In Proceedings of European Conference on Computer Vision (ECCV) , page 436-454, 2020. 2

Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1326-1335, 2021. 2

Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang. Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2021. 2

Shentong Mo and Yapeng Tian. Multi-modal grouping network for weakly-supervised audiovisual video parsing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2022. 2

Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2022. 2

Binglu Wang, Yongqiang Zhao, and Xuelong Li. Multiple instance graph learning for weakly supervised remote sensing object detection. IEEE Transactions on Geoscience and Remote Sensing , 60:1-12, 2021. 3

Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal intervention for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems , 33:655-666, 2020. 3

Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 876-885, 2017. 3

Youngmin Oh, Beomjun Kim, and Bumsub Ham. Background-aware pooling and noise-aware loss for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6913-6922, 2021. 3

Anurag Kumar and Bhiksha Raj. Audio event and scene recognition: A unified approach using strongly and weakly labeled data. 2017 International Joint Conference on Neural Networks (IJCNN) , pages 3475-3482, 2016. 3

Anurag Kumar and Bhiksha Raj. Audio event detection using weakly labeled data. In Proceedings of the 24th ACM International Conference on Multimedia , page 1038-1047, 2016. 3

Ankit Shah, Anurag Kumar, Alexander Hauptmann, and Bhiksha Raj. A closer look at weak label learning for audio events. arXiv preprint arXiv:1804.09288 , 2018. 3

John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 31-35, 2016. 3

Prem Seetharaman, Gordon Wichern, Shrikant Venkataramani, and Jonathan Le Roux. Classconditional embeddings for music source separation. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 301-305, 2019. 3

Fatemeh Pishdadian, Gordon Wichern, and Jonathan Le Roux. Finding strength in weakness: Learning to separate sounds with weak supervision. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:2386-2399, 2019. 3

Efthymios Tzinis, Scott Wisdom, John R. Hershey, Aren Jansen, and Daniel P. W. Ellis. Improving universal sound separation using sound classification. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 96-100, 2020. 3

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6399-6408, 2019. 4, 5

Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling-based design for real-time salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3917-3926, 2019. 5

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. 6

Jia Deng, Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A LargeScale Hierarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 248-255, 2009. 6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new framework for weakly supervised audio-visual segmentation which does not need pixel level annotations. This is achieved via the proposed multi-scale multiple-instance contrastive learning approach which can capture audio-visual alignment in multiple scales. Comparison with existing methods show that the proposed methodology leads to state-of-the-art results.\n\nSTRENGTHS:\nThe paper proposes an interesting approach ti audio-visual segmentation which does not rely on pixel-level annotations. This is an important contribution since obtaining such annotations is time consuming.\n\nThe paper is easy to follow. \n\nConvincing results are presented and a detailed ablation study is presented.\n\nWEAKNESSES:\nThe main weakness of the paper is that several important details are not described.\n\n- In table 1, results for multiple sources are presented without providing details, e.g., how many sources are used, how were these examples obtained? \n\n- It is not clear how the multi-scale features are generated.\n\n- How are the negative examples selected in multi-scale multiple instance contrastive learning? Would be very helpful if the authors elaborate on this.\n\n- Equal weights are used for the two loss terms in Eq. 8. Have the authors considered using different weights? This might improve the performance.\n\n- It is not clear why the face and body are predicted as the sound producing regions in case a person speaks. Ideally, just the face or just the mouth should be identified as the sound source.\n\n- The audio which corresponds to the images in Fig.2 is not presented. Most of the examples are included in the supplementary material so adding such a comment in the caption would be helpful.\n\n- Finally, it would be helpful to add additional failure cases. Only one example is included in the supplementary material. \n\nAnother weakness, is that only one dataset is used for evaluation.\n\nQUESTIONS:\nPlease see the questions above.\n\nLIMITATIONS:\nAs mentioned above, it would be desirable if the authors include additional failure cases in the supplementary material. At the moment there is only one.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new setting for audio-visual segmentation, Weakly-Supervised Audio-Visual Segmentation (WS-AVS). The authors address the challenge of the costly and not always available pixel-level masks by employing weakly-supervised audio-visual segmentation. This framework uses multi-scale multiple-instance contrastive learning for  capturing multi-scale audio-visual alignment. Tested on the AVS Bench, WS-AVS demonstrates superior performance in both single-source and multi-source scenarios when compared to previous methods.\n\nSTRENGTHS:\n1. To eliminate the requirement of pixel-level annotations, this work propose to solve the audio-visual segmentation problem in a weakly-supervised way. Correspondingly, it uses adopts Audio-Visual Fusion with Multi-scale Multiple-Instance Contrastive Learning and Pseudo Mask Refinement by Contrastive Class-agnostic Maps to solve the challenges in the weakly supervised setting.\n\n\n2. The authors provide comprehensive evaluation under various setting to illustrate the performance of the proposed method.\n\nWEAKNESSES:\n1. Clarifying Novelty\nThe paper's novelty looks somewhat unclear. This reviewer recommends clearly defining the unique technical contributions proposed by this work, especially those set apart from the developments in weakly-supervised segmentation. For instance, it would be advantageous to underscore the importance of Multi-scale Multiple-Instance Contrastive Learning in the given task and its distinctiveness from similar concepts previously employed in weakly-supervised segmentation techniques.\n\n2. Further analysis\nThe authors offer sufficient ablation studies to ascertain the performance improvement attributed to AVF and PMR. However, the reviewer anticipates a more in-depth analysis of the outcomes. Specifically, what distinguishes the audio-visual relationship as learned via supervised methods from those learned in weakly-supervised ways? Considering the final classification outcome, it's intuitive to hypothesize that weakly-supervised methods might result in a sub-optimal or noisy audio-visual mapping. But, without human bias, these weakly-supervised maps might demonstrate superior generalization or aid in identifying human annotation-induced preconceptions. The reviewer encourages exploring these insights rather than viewing weakly-supervised audio-visual segmentation merely as a workaround for data limitations.\n\nQUESTIONS:\nThe reviewer believes the task of weakly-supervised audio-visual segmentation is worth exploring, and hence currently vote for borderline accept. However, the authors should adequately discuss the key novelty of this paper for the final acceptance.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new contrastive learning approach with pseudo-mask generation/refinement process for weakly supervised video segmentation with audio guidance. The authors show that their method outperforms several other state-of-the-art methods for the visual segmentation task when training and testing on the AVSBench dataset.\n\nSTRENGTHS:\n- The paper shows strong results compared to other methods in the literature.\n- The paper tries to tackle a very important problem in the vicinity of audio-visual perception which is how to find the pixel regions that correspond to sounding sources when limited annotated data are available.\n- The paper is well written and the ideas are clearly presented.\n\nWEAKNESSES:\nUnfortunately the paper also contains several weaknesses that narrow down the scope and the potential impact of the paper that could be addressed using experiments and rewriting some parts of the manuscript when needed.\n\n- The authors only include experiments with a very small scale in terms of multiple aspects (video frames, classes of sounds only 23 objects, number of videos < 5k) to compare their method with other established methods. Although having smaller scale experiments help with finetuning of the hyperparameters and early experimentation, it becomes difficult to extrapolate the findings of this study to large scale solutions and real-world data with only showing results on the chosen dataset. Some dataset that come on top of my head and could be used by the authors to enrich the results and show the true potential of their method would be some more limited datasets like Flickr-SoundNet [A], a medium-sized mostly single-source dataset like VGG Sound Source [B] or a real-world video dataset like AudioSet [C] or YFCC100M [D].\n- Building upon my previous argument, using more realistic and large scale datasets would not only put away potential skepticism revolving around the validity of training neural networks with only 5k examples and limited video resolutions (e.g. capturing motion features when the average video has 2 frames is almost impossible for the dataset used in this paper) but would also show whether the proposed approach is robust when trained with videos that contain no on-screen objects which is very difficult to solve for related problems like on-screen sound separation [E]. I don’t want to extend the problem to cases where more than one sounds appear on-screen / off-screen and thus some audio separation front-end might be needed before obtaining the audio features but I would still expect some more convincing large-scale experiments.\n- In the current version of the paper, the authors only consider evaluation datasets where the sounding object appears on-screen at some point in the video but they completely omit to report the performance for off-screen-only videos. To that end, the authors should also evaluate the performance of their model when there is some visual activity on-screen but the sounding object/action cannot be seen in camera and test how robust their model is. According to my experience, there will be a trade-off in the sensitivity of how accurate the model will predict the masks for on-screen objects which correspond to the on-screen audio and how well the model will predict zeros for on-screen objects and actions that do not make any sound (e.g. a human talking in the background while a non-talking human is displayed on-screen).\n\nOverall, I am willing to increase my score if most of the most important above concerns are addressed by the authors since I truly believe that the paper has a great potential to become a good benchmark method for weakly supervised audio-guided video segmentation.\n\n[A] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018.\n\n[B] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020.\n\n[C] Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M. and Ritter, M., 2017, March. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 776-780). IEEE.\n\n[D] Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D. and Li, L.J., 2016. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2), pp.64-73.\n\n[E] Tzinis E, Wisdom S, Jansen A, Hershey S, Remez T, Ellis D, Hershey JR. Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds. In International Conference on Learning Representations 2021.\n\nQUESTIONS:\nHave you considered an analysis of the proposed approach for the choice of the model for ensuring robustness towards the “reliable pseudo mask extraction with multi-scale visual features”?\n\nLIMITATIONS:\nThe authors discuss some limitations of their work and I think they should include the issues that I raised in the Section above and that they will not be able to address through experiments and/or rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a weakly-supervised audio-visual segmentation framework called WS-AVS  that predicts sounding source masks from audio and images without pixel-level ground truth masks. It leverages multi-scale contrastive learning in audio-visual fusion to capture multi-scale alignment, addressing modality uncertainty in weakly supervised segmentation. The refined pseudo masks guide training, enabling generating accurate segmentation masks. Experiments show it outperforms weakly-supervised baselines.\n\nSTRENGTHS:\nThe manuscript presents a solution to the challenging problem of performing segmentation tasks without pixel-level annotations by effectively integrating multi-scale contrastive learning and pseudo mask refinement. It is well written and easy to read through.\n\nWEAKNESSES:\n1. Although these methods have been proven effective independently, their combination in this manuscript does not elevate them to a novel plane but rather presents a blend of established tactics, the novelty aspÅect therefore seems insufficient.\n\n2. While the max-pooled cosine similarity function works well in [9], is there any other similarity function that can be compared with, as AVS has multi-source scenarios?\n\n3. As the WS-AVS framework relies heavily, to some extent, on the generation of pseudo-labels, so is there any other ablations on pseudo-label generation?\n\n4. Noticed that there is no schematic for multi-source in the manuscript, wondering what the pseudo-label for multi-source looks like.\n\n5. In Line 188-190, the authors wrote: the recent audio-visual segmentation baseline [1] does not give any cross-modal constraint on the audio and visual representations in the audio-visual fusion, which causes a significant performance drop when removing the ground-truth masks during training. I notice that [1] proposed an audio-visual matching loss that is used in the multiple sound source case as [1] claimed this loss does not bring further improvement in single-source case. I guess the pixel-level ground truth provides enough supervision in single-source AVS. However, in the studied weakly-supervised AVS setting, will this loss help in the audio-visual fusion?\n\n5. Some minor issues:\n\nTypo. Line 54, “WS-VAS, that” should be “WS-AVS, which”\n\nTypo. Line 176, “V^s” should be “V^s_i”\n\nTypo.  Line 194 “WSSL” should be “WSSS”\n\nThe result in F-score of Multiple Source in Table 1 should be bold.\n\nQUESTIONS:\nAs above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}