Weakly-Supervised Audio-Visual Segmentation
Authors: Shentong Mo
Bhiksha Raj
Venue: NeurIPS 2023 poster (2023)
Keywords: audio-visual learning
visual sound localization
audio-visual segmentation

Abstract: Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video.
  Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision.  However, these pixel-level masks are expensive and not available in all cases.
  In this work, we aim to simplify the supervision as the instance-level annotation, $\textit{i.e.}$, weakly-supervised audio-visual segmentation.
  We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation.
  Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.

Weakly-Supervised Audio-Visual Segmentation
Shentong Mo! Bhiksha Raj
'CUU, MBBZAAI
Abstract
Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as su- pervision. However, these pixel-level masks are expensive and not available all cases. In this work, we aim to simplify the supervision the instance-level annotation, i.e. weakly-supervised audio-visual segmentation. We present a novel Teakky-SSuerviise Audio-Visur Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVS- Bench demonstrate the effectiveness of our WS-AVS in the superls-e audio-visual segmentation of single-source and multi-source scenarios.
Introduction
When we hear a dog barking, we are naturally aware of where the dog is in the room due to the strong correspondence between audio signals and visual objects in the world. This human perception intelligence attracts many researchers to explore audio-visual joint learning for visual sound localization and segmentation. In this work, we aim segment sound source masks from both frames and audio in a video without relying on expensive annotations of pixel-level masks, ie., only the instance-level annotation is given for training.
Audio-visual segmentation (AVS) [1] a recently rising problem that predicts a pixel-level map of objects that produce sound at the time of frames. A related problem to this recent task is sound source localization, which aims to locate visual regions within the frame that correspond to the sound. Sound source localization (SSL) [2, 3, 4, 5, 6, 7, 8, 9, 10] estimates rough location of the sounding objects at a patch level, while the goal of AVS is to estimate pixel-wise segmentation comprehensive manually designed architecture
masks for all sound sources. AVS requires than SSL since it needs to identify the exact shape of sounding objects. However, this complicated network for AVS requires countless pixel-wise accurate masks as supervision, which costs numerous labor and resources. In contrast, we aim to get rid of these costly pixel-level annotations to achieve weakly-supervine
audio-visual segmentation.
Many supervise methods [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] have been proposed to be successful in the visual segmentation community, where they replaced accurate pixel-level annotations with weak labels, including scribbles [13, 15], bounding boxes [1 12], points [14), and image-level class labels [16, 17]. Typically, CAM [18] was proposed with CNN-based image classifier to generate pseudo segmentation masks at the pixel level using coarse object localization maps. With coarse localization maps, following works tried obtain the full extent of objects by regions expansion [21, 22, 23], stochastic inference [24], boundary constraints [25, 26, 27], or object regions mining and erasing [28, 29]. However, only one modality, i.e., image is involved in these aforementioned CAM-based weakly-supervised approaches. In this work, we need to tackle with two distinctive modalities (audio and vision), where well-designed cross- modal fusion is required to aggregate indio-visual features with high similarity in the semantic space.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The main challenge is that we do not have pixel-level segmentation masks for this new weakly- supervised multi-modal problem during training. Without this supervision, the performance of AVS [1] on audio-visual segmentation degrades significantly, observed in our experiments in Section 4.2. the meanwhile, previous weakly-supervised semantic segmentation [18, 30] and visual sound source localization [9] methods have two main challenges. First, no multi-modal constraint is applied for pixel-wise mask prediction in the previous weakly-supervised semantic segmentation methods as they do not use the audio input during training. Second, current visual sound source localization baselines generate only coarse heatmaps for sounding objects instead of binary pixel- wise segmentation masks. To address the aforementioned challenges, our key idea is to apply multi-scale multiple-insten instance contrastive learning in audio-visual fusion to learn multi-scale cross- modal alignment, which differs from existing weakly-supervised semantic segmentation approaches. During training, we aim to use a reliable pseudo mask with multi-scale visual features as mask-level supervision for predicting accurate audio-visual segmentation masks in weakly-supervised setting.
To this end, we propose a novel seakky-superiieed audio-visual segmentation framework, namely WS-VAS, that can predict sounding source masks from both audio and image without using the ground-truth masks. Specifically, our WS-AVS leverages multi-scale multiple-instance contrastive learning in audio-visual fusion to capture multi-scale cross-modal alignment for addressing modality uncertainty in by contrastive class-agnostic maps will serve pixel-level guidance during training. Compared to previous visual sound source localization approaches, our method can generate binary accurate audio-visual segmentation masks.
semantic segmentation methods. Then, the pseudo mask refined
Empirical experiments on AVSBench [1] comprehensively demonstrate the state-of-the-ar per- formance against previous weakly- baselines. In addition, qualitative visualizations of segmentation masks vividly showcase the effectiveness of our WS-AVS in predicting sounding object masks from both audio and image. Extensive ablation studies also validate the importance of audio- visual fusion with multi-scale multiple-instance contrastive learning and pseudo mask refinement by contrastive class-agnostic maps in single-source and multi-source audio-visual segmentation.
Our main contributions can be summarized as follows:
We investigate a new weakly-superviset multi-modal problem that predicts sounding object masks from both audio and image without needing pixel-level annotations.
We present a novel framework for weakly-supervised audio-visual segmentation, namely WS-AVS, with multi-scale multiple-instance contrastive learning to capture multi-scale audio-visual alignment.
Extensive experiments comprehensively demonstrate the state-of-the-art superiority of our WS-AVS over previous baselines single-source and multi-source sounding object segmentation.
2 Related Work
Audio- Visual Learning. Audio-visual learning has been explored in many existing works [31, 32, 33, 34, 2, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46] to learn audio-visie correspondence from those two distinct modalities in videos. Given video sequences, the goal is to push away those from different pairs while closing audio and visual representations from the same pair. Such cross-modal correspondence is useful for several tasks, such as audio/speech separation [37, 47, 48, 35, 36, 49, 50, 51, 52], visual sound source localization [2, 3, 4, 5, 6, 7, 8, 9, 10, 53, 54], audio spatialization [55, 56, 38]. and vidiul parsing [58, 59, 60, 61]. In this work, we mainly focus on lcarning audio-visual association for pixel-level audio-visual segmentation, which is more demanding than those tasks mentioned above.
Visual Sound Localization. Visual sound source localization aims to identify objects or regions of a video that correspond to sounds the video. Recent researchers [2, 4, 5, 7, 8, 62, 9, 10] used diverse networks to learn the audio-visual alignment for predicting sound sources in the video. Typically, Attention 10k [2] proposed a two-stream architecture based on an attention mechanism to detect sounding objects in the image. More recently, EZVSL [9] applied a multiple-instance contrastive learning objective to learn the alignment across regions with the most corresponding audio. However, they cannot predict exact masks for sounding objects.

---

Audio- Visual Segmentation. Audio-visual segmentation is a challenging problem that predicts pixel- wise masks for sounding objects in the image. In order to address this issue, Zhou et al. [1] introduced the first audio-visual segmentation benchmark with pixel-level annotations for each 5-second video, where a binary mask is used to indicate the pixels of sounding objects for the corresponding audio. Moreover, they proposed an encoder-decoder network with a temporal pixel-by-pixe audio-visual interaction module to solve the pixel-level audio-visual segmentation task. However, their approach requires exhausted pixel-level annotations the main supervision. Without this supervision, their performance degrades significantly as observed in our experiments in Section 4.2. In this work, we aim to get rid of the dependency of pixel-level labels, by leveraging only instance- labels for weakly-superviser audio-viss visual segmentation.
Weaaly-Supervvsed Semantic Segmentation. Weakly-supervised visual segmentation aims at reduc- ing the burden of collecting pixel-level annotations for image segmentation at a large scale required by its fully-supervised counterpart. Early methods explored different forms of weak supervision, including image-level label [63, 64], scribble [13, 15], and bounding box [12, 65, 66]. Typically, CAM [18] was introduced to generate class activation maps as pseudo pixel-level segmentation masks for weakly
object localization. Based on CAM, following works tried to optimize the coarse class activation map for more precise object localization, by expanding localization re- gions [21, 22, 23], using stochastic inference [24]. exploring boundary constraints [25, 26, 27], or mining and erasing regions of objects [28, 29]. More recently. C2 AM [30] leveraged contrastive learning with the semantic relation between foreground and background as positive and negative pairs to generate class- -lassaggnostic activation map. Different from weakly-supervised visual segmentation baselines, we aim to develop a fully novel framework to aggregate semantics from both audio and visual representations with the only instance-level annotation. To the best of our knowledge, we the first to explore class-agnostic activation maps on weakly-supervised audio-visual segmentation. In addition, we leverage audio-visual multi-scale fusion to boost the segmentation performance.
Audio Learning. Weakly-supervised learning has been widely-used in many audio-relevant tasks, such as sound events detection [67, 68, 69] and sound source separation [70, 71, 72, 73]. Typically, WAL-Net [69] proposed using weakly labeled data from the web to explore how the label density and corruption of labels affect the generalization of models. DC/GMM [71] applied an auxiliary network to generate the parameters of Gaussians in the embedding space with a one-hot vector indicating the class as input. However, the problem need to solve in this work is more difficult and challenging. Indeed, we aim to leverage the weakly labelled supervision of image categories solve the new veakly-supprrvsee multi-modal problem.
3 Method
Given clip of audio and video frames, our target is to predict pixel-level masks for sounding objects in the frame. We present novel Weakly-Supervised Audio-Visus Segmentation framework named WS-AVS for single source audio-visual segmentation without involving the pixel-wise annotation, which mainly consists of two modules, Audio Visual Fusion with Multi-scale Multiple Instance Contrastive Learning in Section 3.2 and Pseudo Mask Refinement by contrastive class-agnostic maps in Section 3.3.
3.1 Preliminaries
In this section, we first describe the problem setup and notations, and then revisit the first work [1] with pixel-level annotations for single source audio-visual segmentation.
Problem Setup and Notations. Given an audio spectrogram and an image, our goal is to predict the binary segmentation mask for the sound source in the image spatially. Let D (u,, v.) : = 1,.., N be a dataset of paired audio 0; E R and visual data 0 E xx where only one sound source a, are assumed to be existing in U. Note that T, F denotes the dimension of time and frequency of the audio spectrogram. We follow previous work [1, 9] and first encode the audio and visual inputs using two-stream neural network encoder and projection heads, denoted as fa ga (-) and fe()), (-) for the audio and images, separately. The audio encoder computes global audio representations a; 9o (ax))) a; RR IIID and the visual encoder extracts multi-scale representations {v;) (v.)) vf e R Dx W for each sth stage. During the training, we do not have lask-level annotations Y E R HxW
3

---

Other Image
Image
Audio-Visusi Fuslon
Vin
negalive
Multi-scale Multiple Instance Contrastive Loss
Audio
Meow
Audio Encoder
global audio feature
postive
Audio-Visual Fusion
M
Decoder
mask prediction
Pixel-wise Loss
Image
Pseudo Mask Refinement
Encoder
multi-scale visual features
Figure Illustration the proposed Weakly-Superviset Audio Visual Segmentation (WS-AVS) framework. Global audio features a, and multi-scale visual features {v:) are extracted from the audio spectrogram and image via an audio encoder and an image encoder, respectively. The Ludio-Visure Fusion module combines multi-scale multiple-instance contrastive learning to capture multi-scale cross-modal alignment between global audio and multi-scale visual features. Then, with multi-scale visual features {v:] s=1, / the Pseudo Mask Refinement module generates reliable pseudo mask Y for sounding objects by using contrast class-agnostic map. Finally, decoder is applied on multi-scale audio-visual features {z, } to predict masks. The pixel-wise loss is simply optimized between the predicted mask M and the pseudo mask Y.
Revisit Audio-Visual Segmentation with Pixel-level Annotations. To solve the single source audio-visual segmentation problem, AVS [1] introduced the pixel-wise audio-visure visual fusion module to encode the multi-scale visual features and global audio representations. After the cross-modal fusion, the audio-visual feature map z at the sth stage is updated as:
(((v))((a)) H-W* w(v)))
2 = v +
where x denotes the duplicated version of the global audio representation a, that repeats H"W times at the sth stage. μu-). (()), (0-),,())) w(-),w(-) denote the x 1 x convolution operator. Then those updated multi-stage feature maps are passed into the decoder of Panoptic-FPN [74] and sigmoid activation layer to generate the final output mask M E I #x With the pixel-level annotation Y as supervision, they applied the binary cross entropy loss between the prediction and label as:
baseline BCE(M, Y)
However, such training mechanism is extremely dependent on pixel-level annotations. Without this supervision, their performance on single source audio-visual segmentation deteriorates significantly, as observed in Section 4.2. In the meanwhile, current weakly-supervised semantic segmentation (WSSS) [18, 30] and visual sound source localization (VSSL) [9] methods will pose two main challenges. First, there is multi-modal constraint for pixel-level mask prediction in the previous WSSS methods as they do not involve the audio during training. Second, existing VSSL approaches only predict coarse heatmaps for sounding sources instead of accurate pixel-wise segmentation masks. To address these challenges, we propose a novel seakly-supervised audio-visure segmentation framework for single sounding object segmentation, which is composed of Audio Visual Fusion with multi-scale multiple-instance contrastive learning and Pseudo Mask Refinement by contrastive class-agnostic map, shown in Figure 1.
3.2 Audio- Visual Fusion with Multi-scale Multiple-Instance Contrastive Learning
In order to address the modality uncertainty brought by the previous WSSS baselines [18, 30], inspired by EZ-VSL [9], propose to conduct audio-visual fusion by focusing only the most

---

aligned regions when matching the audio to the multi-scale visual feature. This is because most locations in the video frame do not correspond the sounding object, and thus the representations at these locations should not be aligned with the audio during training.
To align the audio and visual features at locations corresponding sounding sources, we apply the multi-scale multiple-instanee contrastive learning IM'ICL) objective to align at least one location in the corresponding bag of multi-scale visual features with the audio representation the same mini-batch, which is defined as:
B
S
exp ( simm(a, v:))
fu-e B
log
0==1 exp (imim a vi))
1=1 ==1
where the similarity sim(a;, a, v²) denotes the max-pooled vudio-visual cosine similarity of a, and v? across all spatial locations at stage. That is, sim(a,, vf) max vf). B is the batch size, D is the dimension size, and r a temperature hyper-parameter.
Similar to EZ-VSL [9], we use a symmetric loss of Hq. 4 to discriminate negative audio bags from other audio samples in the same mini-batch, which is defined as
1
B
9
exp (+sim(a, a vi))
C L
log
B
m exp
s ams v())
where am denote the global audio visual from other sample m in the mini-batch. The overall audio- visual fusion objective with the multi-scale multiple-instance contrastive learning mechanism defined as:
B
exp simia, vf))
Lvv = B
log
+
m=1 exp simmai, vin))
(5)
exp (isima v:))
log
exp sim(am v())
Optimizing the loss will push the model learn discriminatively global audio representation a, and multi-scale visual features {v]}s=] Then, these features are fed forward into Eq. to generate updated multi-scale audio-visual features {zī )= Finally, with these updated multi-scale audio- visual features, we apply the decoder of Panoptic-FPN [74] to generate the output mask M. Note that, the recent audio-visual segmentation baseline [1] does not give any cross-modal constraint on the audio and visual representations in the audio-visual fusion, which causes significant performance drop when removing the ground-truth masks during training.
3.3 Pseudo Mask Refinement by Contrastive Class-agnostic Maps
The second challenge of spatial uncertainty VSSL, approaches [9] requires us to utilize mask-level supervision for generating precise final output mask M. To generate reliable pseudo masks of training sets, motivated by recent WSSL pipelines [30], we introduce the Pseudo Mask Refinement by applying a contrastive class-agnostic map on multi-scale visual features {vī)š Specifically, we utilize the contrastive loss in [30] close the distance between the representations in positive pairs wwn.grounddfooegrouun
and push away the representations in negative pairs (foreground- -aggrouundbackgrouud) Then, we use the background activation maps as pscudo labels to further train a salient object detector [75] to predict the salient region S R 1 W in the image. Finally, the predicted salient cues are concatenated with the initial class-agnostic map A E D to perform the argmax operator along the embedding dimension to generate the binary pseudo mask Y I HW which is formulated as:
background-backgee
Y L arg max A)) I+D
(6)
where [: denotes the concatenation operator. Note that since we are focusing on single-source segmentation, L [Ls: LA] (1+)))XX> Lg E x is the full label of zeros for the salient cues, while LA E R xxx is the full label of ones for the class-agnostic maps. With the
5

---

Table 1: Comparison results (%) of weakly-supervie mudio-visual segmentation. "ws" denotes the weakly-supervisse baseline, where only the instance- class is used for training.
Single Source
Multiple Source
Method
mloU (f)
F-score (1)
mloU (1)
F-score (1)
AVS [1] (ws)
12.63
24.99
8.76
15.72
CAM [18]
19.26
27.88
12.65
19.83
EZ- EZL [91
29.40
35.70
23.58
27.31
C2AM [30]
30.87
36.55
25.33
29.58
WS-AVS (ours)
34.13 (+3.26)
51.76 (+15.21)
30.85 (+5.52)
46.87 (+17.29)
reliable pseudo mask refinement, we apply the binary cross entropy loss between the predicted mask M and the pseudo mask Y, which is defined as:
Ppmr = BCB(M, Y)
(7)
The overall objective of our model is simply optimized in an end-to-end manner as:
C Lavf + Cpmr
(8)
During inference, we follow previous work [1] and directly use the predicted mask M E R #x as the final output. It is worth noting that the final localization map in the current VSSL methods [9] was generated through bilinear interpolation the audio-visual feature map at the last stage.
Experiments
4.1 Experiment Setup
Datasets. AVSBench [1] contains 4,932 videos with 10,852 total frames from 23 categories including animals, humans, instruments, etc. Following prior work [1], we use the split of 3,452/740/740 videos for train/val/test in single source segmentation.
Evaluation Metrics. Following previous work [1], we the averaged IoU (mloU) and F-score to evaluate the audio-visual segmentation performance. mloU computes the intersection-over-unioo (IoU) of the predicted mask and ground-truth mask for evaluating the region similarity. F-score calculates both the precision and recall for evaluating the contour accuracy.
Implementation. The input image is resized to a resolution of 224 x 224. The input audio takes log spectrograms extracted from 3s of audio at a sample rate of 22050Hz. Following previous works [9, 10]. we apply STFT to generate
input tensor of size 257 300 (257 frequency bands over 300 timesteps) using 50ms windows with hop size of 25ms. We follow the prior audio-visual segmentation work [1] and use the ResNet50 [76] as the audio and visual encoder. The visual model initialized using weights pre-trained on ImageNet [77]. The model is trained with the Adam optimizer with default syper-paramedt
0.9, B2 0.999, and learning rate of le-4. The model trained for 20 epochs with a batch size of 64.
4.2 Comparison to Prior Work
In this work, we propose a novel and effective framework for weakly audio-visual segmentation. To validate the effectiveness the proposed WS-AVS, we comprehensively compare it to previous audio-visual segmentation, weakly-supervised semantic segmentation, and visual sound source localization baselines: 1) AVS [1](ws): the weakly-supervised version of the first audio-visual segmentation work, where we removed the pixel-level annotations for training; 2) CAM [18]: the first baseline using activaaiii maps as pseudo segmentation masks for weakly-supervised object localization; 3) EZ-VSL [9]: the state-of-the-e
visual sound source localization approach with coarse source maps as output; 4) C² AM [30]: the very recent work for -uakly--upervieed semantic segmentation with only the image as input.
The quantitative comparison results are reported Table can seen, we achieve the best performance in terms of all metrics compared to previous weakly-supervie baselines. In particular,

---

Raw Image
Ground-Truit
AVS (ws)
CAM
EZ-VSL
WS-AVS (ours)
Figure 2: Qualitative comparisons with
audio-visual segmentation (AVS [I]) wreakly-superviag semantic segmentation (CAM [18], C² AM [30]), and visual sound source local- ization (EZ-VSL [9]) baselines on single-source segmentation. The proposed WS-AVS produces much more accurate and high-quality segmentation maps for sounding objects.
the proposed WS-AVS significantly outperforms AVS [1](ws), the weakly-supervinn version of the state-of-the-art audio-visue segmentation approach, by 21.50 mloU and 26.77 F-score. Moreover, we achieve superior performance gains of 3.26 mloU and 15.21 F-score compared to C²AM [30], which implies the importance of the proposed multi-scale multiple-instance contrastive learning for addressing the modality uncertainty in audio-visual fusion. Meanwhile, our WS-AV' outperforms EZ- VSL [9], the current state-of-the- art visual sound source localization baseline, by a large margin, where achieve the performance gains of 4.73 mloU and 16.06 F-score. These significant improvements demonstrate the superiority of our method in single-source audio-visual segmentation.
In addition, significant gains in multi multi-source sound localization can be observed in Table 1. Compared version of the state-of-the-ar audio-visual segmentation
to AVS [1](ws) the approach, achieve the results gains of 22.09 mloU and 31.15 Fl score. Furthermore, the proposed approach still outperforms C² AM [30] by 5.52 mloU and 17.29 score. We also achieve highly better results than EZ-VSL [9], the current state-of-the-art visual sound source localization baseline. These results validate the effectiveness of our approach in learning multi-scale multiple-instane semantics from mixtures and images for multi-source localization.
In order to qualitatively evaluate the localization maps, we compare the proposed WS-AVS with weakly-supervicn AVS [1], CAM [18], CM [30], and EZ-VSL [9] single-source segmentation Figure 2. From comparisons, three main observations can be derived: 1) Without pixel-level annotations, AVS [1], the first audio-visus segmentation fails to predict the masks for sounding objects; 2) the quality of segmentation masks generated by our method is much better than the strong visual sound source baseline, EZ-VSL [9]; 3) the proposed WS-AVS achieves competitive even better results on predicted masks against the weakly-supervised semantic segmentation baseline [30] by using contrast class-agnostic maps for prediction. These visualizations further showcase the superiority of our simple WS-AVS with multi- scale multiple-instant contrastive learning to guide segmentation for predicting accurate source masks.
4.3 Experimental Analysis
In this section, we performed ablation studies to demonstrate the benefit of introducing the Audio- Visual Fusion with multi-scale multiple-instance contrastive learning and Pseudo Mask Refinement module by contrastive class-agnostic map. We also conducted extensive experiments to explore the effect of fusion stages and batch size weakly supkrv-s audio-visua segmentation. Furthermore,
7

---

Table 2: Ablation studies Audio-Visual Fusion (AVF) and Pseudo Mask Refinement (PMR).
AVF
PMR
mloU
F-score (1)
X
X
12.63
24.99
V
X
31.85 (+19.22)
39.78 (+14.79)
X
V
32.46 (+19.83)
43.92 (+18.93)
/
/
34.13 (+21.50)
51.76 (+26.77)
68
60
F-score
Fiscare
20
32
Fusion Stage (S)
Batch Size (B)
Figure 3: Effect of fusion stages and batch size on mloU and F-score.
we visualized high-quality pseudo masks from Pseudo Mask Refinement in weakly-supervised training.
Audio-Visual Fusion & Pseudo Mask Refinement. To validate the effectiveness of the introduced audio-visual fusion with multi-scale multiple-instance contrastive learning (AVF) and Pseudo Mask Refinement by contrastive class-agnostic map (PMR), we ablate the necessity of each module and report the quantitative results in Table 2. We can observe that adding AVF to the vanilla baseline highly increases the results of single-source audio-visual segmentation by 19.22 mloU and 14.79 F-score, which demonstrates the benefit of multi-scale multiple-instance contrastive learning (M*ICL) in extracting aligned cross-modal feature in fusion for source segmentation. Meanwhile, introducing only PMR in the baseline also increases the segmentation performance in terms of all metrics. More importantly, incorporating AVF with ICCL and PMR together into the baseline significantly raises the performance by 21.50 mloU and 26.77 F-score. These improving results validate the importance of AVF with IC and PMR with contrastive agnnssic map in addressing both modality and spatial challenges for generating accurate audio-visua masks.
Effect of Fusion Stages and Batch Size. The number of fusion stages and batch size used in the proposed MCCL affect the extracted cross-modal representations for audio-visual segmentation. To explore such effects more comprehensively. we varied the number of fusion stages from {1, 2, 3, 4) and ablated the batch size from {8, 16, 32, 64, 128). The comparison results of segmentation perfor- mance are shown in Figure 3. When the number of fusion stages
and using batch size of 64 in I'ICL, we achieve the best segmentation performance in terms of all metrics. With the increase of fusion stages from 4, the proposed WS-AVS consistently raises results, which shows the impor- tance of multi-scale visual features in audio-visual fusion for learning discriminative cross-modal representations. Regarding the batch size, the performance of the proposed WS-AVS climbs with the increase of the batch size from 8 to 64. However, increasing the batch size from 64 to 128 will not continually improve the result since there might be some false negatives in the mini-batch for this training set with a relatively smaller size.
Generated Pseudo Mask. Generating reliable pseudo masks with contrast class-agnostic maps is critical for us to train the weakly-supervine pseudo masks, we visualize the pseudo mask and Mlass-agnosti map Figure As can be observed in the last column, the class-agnostic map can successfully localize the sounding object in the image. Furthermore, the pseudo masks in the third column predicted by Eq. 6 have high quality and they are even fine-grained compared to the manually annotated ground-truth masks. These meaningful visualization results further showcase the success of the Pseudo Mask Refinement in extracting reliable pseudo masks as guidance for songle-source audio-visual segmentation.
framework. To better evaluate the quality of generated

---

Gronnd-rrum
Row Image
Poeado Mask
Clasappodte Map
Ground Torh
Raw Enage
Perodo Mask
Chasoggodi Map
Figure 4: Qualitative comparisons of the ground-truth mask, pseudo mask, and class-agnostic map. Note that the generated pseudo mask is used for weakly-supervised training.
4.4 Limitation
Although the proposed WS-AVS achieves superior results on single-source audio-visne visual segmentation, the performance gains our approach on the mloU metric are not significant. One possible reason is that our model easily overfits across the training phase, and the solution is to incorporate dropout and momentum encoders together for weakly-supervised audio-visual segmentation. Meanwhile, we notice that our model performs worse mixed sound sources, such scenario with both female singing and playing tabla. The future work could be to add separation objectives or assign semantic labels to both audio and visual segments in audible videos.
Conclusion
audio visual segmentation to eliminate pixel-level annotations supervision. Then present WS-AVS, novel Weakly-Superrised Audio- Visual Segmentation framework with multi-scale multiple-instance contrastive learning for capturing multi-scale audio-visual alignment. Furthermore, we conduct extensive experiments AVSBench to demonstrate the superiority of our WS-AVS against previous weakly-supervised audio- visual segmentation, weakly-supervised semantic segmentation, and visual sound source localization approaches.
In this work, we successfully investigate the
Broader Impact. The proposed approach successfully predicts segmentation masks of sounding sources from manually-collecte datasets on the web, which might cause the model learn internal biases in the data. For instance, the model could fail to discover rare but crucial sound sources. Therefore, these issues should addressed for the deployment of real applications.
References
[1] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference Computer Vision, 2022. 2, 3, 4. 6,7
[2] Arda Senocak, Tae- Oh, Junsik Kim, Ming Hsuan Yang, and In So Kweon. Learning to localize sound source visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4358-4366, 2018. 1,2
[3] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh H. McDermott, and Antonio Torralba. Self-supervised audio-visual co-segmentation. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2357-2361, 2019. 1,2
[4] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 9248-922 9257, 2019. 1,2

---

[5] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In Proceedings of European Conference on Computer Vision (ECCV), pages 208 224, 2020. 1,2
[6] Janani Ramaswamy and Sukhendu Das. See the sound, hear the pixels. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2959-2968, 2020. 1,2
[7] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Proceedings of European Conference on Computer Vision (ECCV), pages 292-308, 2020. 1,2
[8] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16867-16876, 2021. 1,2
[9] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Proceedings of European Conference on Computer Vision (ECCV), page 218-234, 2022. 1, 3, 4, 6,
[10] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Proceedings of Advances in Neural Information Prucessing Systems (NeurIPS), 2022. 2,
[11] George Papandreou, Liang-Chich Chen, Kevin P Murphy, and Alan L Yuille. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation, In Proceedings of the IEEE international conference computer vision, pages 1742-1750, 2015.
[12] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In Proceedings of the IEEE international conference on computer vision, pages 1635-1643, 2015. 1,3
[13] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3159-3167, 2016. 1,3
[14] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In European conference computer vision, pages 549-565. Springer, 2016.
[15] Paul Vernaza and Manmohan Chandraker. Learning random-walk label propagation for weakly- supervised semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7158-7166, 2017. 1,
[16] Qibin Hou, Daniela Massiceti, Puneet Kumar Dokania, Yunchao Wei, Ming-Ming Cheng, and Philip HS Torr. Bottom-up top-down cues for weakly-supervised semantic segmentation. In International Workshop on Energy Mininization Methods in Computer Vision and Pattern Recognition, pages 263-277. Springer, 2017.
[17] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568 1578, 2017.
[18] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2922222 2016. 1, 2, 3, 4, 7
[19] Tianfei Zhou, Wenguan Wang, Si Liu, Yi Yang, and Luc Van Gool. Differentiable multi- granularity human representation learning for instance-aware human semantic parsing. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1622-1631, 2021.
[20] Jiapei Feng, Xinggang Wang, and Wenyu Liu. Deep graph cut network for weakly-supervise semantic segmentation. Science China Information Sciences, 64(3):1-12, 2021.
10

---

[21] Alexander Kolesnikov and Christoph H Lampert. Seed, expand and constrain: Three principles for weakly-supervised image segmentation. In European conference on computer vision, pages 695-711, 1. Springer, 2016. 3,
[22] Zilong Huang, Xinggang Wang, Jiasi Wang. Wenyu Liu, and Jingdong Wang. Weakly- supervised semantic segmentation network with deep seeded region growing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7014-7023, 2018. 1,3
[23] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jic, Jiashi Feng, and Thomas S Huang. Revisiting dilated convolution: A simple approach for weakly-and semi-supervised seman- tic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7268-7277 2018. 3,3
[24] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5267-5276, 2019. 1,3
[25] Wataru Shimoda and Keiji Yanai. Self-supervised difference detection for weakly-supervised semantic segmentation. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 5208-5217, 2019. 1,3
[26] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12275-12284, 2020. 1,3
[27] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang. Weakly-supervised semantic segmentation via sub-category exploration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1991-9000) 2020. 3,3
(28) Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 9215-9223, 2018. 1,3
[29] Qibin Hou, PengTao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-erasing network for integral object attention. Advances in Neural Information Processing Systems, 31, 2018. 1.
[30] Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen. C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 989-998, 2022. 2,3, 4,5, 6,7
[31] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016. 2
[32] Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In Proceedings of the European Conference Computer Vision (ECCV), pages 801-816. 2016.
[33] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 609-617, 2017.
[34] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from superved synchronization. In Proceedings of Advances in Neural Information
Processing Systems (NeurIPS), 2018. 2
[35] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 570-586, 2018. 2
11

---

[36] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1735-1744, 2019. 2
[37] Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (0478-1045) 2020. 2
[38] Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In Proceedings of Advances in Neural Information Processing Systems
(NeurIPS), pages 4733 4744, 2020. 2
[39] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12934-12945, 2021. 2
[40] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 12475-12486, June 2021. 2
[41] Shentong Mo and Yapeng Tian. Semantic-aware multi-modal grouping for weakly-supervise audio-visual video parsing. In European Conference Computer Vision (ECCV) Workshop. 2022. 2
[42] Shentong Mo and Pedro Morgado. Benchmarking wcakly-supervised localization. In European Conference Computer Vision (ECCV) Workshop, 2022. 2
sound
[43] Shentong Mo, Jing Shi, and Yapeng Tian. DiffAVA: Personalized text-to-audio generation with
visual alignment. arXiv preprint arXiv:2305. 12900,, 2023. 2
[44] Shentong Mo and Pedro Morgado. A unified audio-visual learning framework for localization, separation, and recognition. arXiv preprint arXiv:2305. 2023. 2
[45] Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. arXiv preprint arXiv:2308./ 2023.
[46] Shentong Mo, Weiguo Pian, and Yapeng Tian. Class inssemnnceeental grouping network for continual audio-visual learning. 2023. 2
[47] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-53, 2018. 2
[48] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3879-3888, 2019.
2
[49] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look; Action recognition by previewing audio. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10457-10467, 2020. 2
[50] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 2745-2754, 2021. 2
[51] Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15495-15505, 2021. 2
[52] Tanzila Rahman and Leonid Sigal. Weakly-supervised audio- visual sound source detection and separation. In 2021 IEEE International Conference Multinedia and Expo (ICME), pages 1-6. IEEE, 2021.
[53] Shentong Mo and Yapeng Tian, Audio-visual grouping network for sound localization from mixtures. arXiv preprint 1735: 2023. 2
12

---

[54] Shentong Mo and Yapeng Tian. AV-SAM: Segment anything model meets audio-visual local- ization and segmentation. arXiv preprint arXiv:2305.01836, 2023. 2
[55] Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for video. In Proceedings of Advances in Neural Information
Processing Systems (NeurIPS), 2018. 2
[56] Ruohan Gao and Kristen Grauman. 2.5d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 324-333, 2019.
[57] Changan Chen, Unnat Jain, Carl Schissler, S. V. A. Garí, Ziad AI-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Proceedings of European Conference on Computer Vision (ECCV), pages 17-36. 2020. 2
[58] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised
video parsing. In Proceedings of European Conference on Computer Vision (ECCV), page 436-454, 2020. 2
[59] Yu Wu and Yi Yang. Exploring heterogeneous clues for weakly- weakly-supervised audio-visual video parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1136-13335 2021. 2
[60] Yan-Bo Lin, Hung- Yu Tseng, Hsin- Ying Lee, Yen- Yu Lin, and Ming-Hsuar Yang. Exploring cross-video and cross-modality signals for weakly-supervised audio-visual video parsing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
[61] Shentong Mo and Yapeng Tian. Multi-modal grouping network for weakly-supervised audio- visual video parsing. In Proceedings of Advances in Neural Information Processing Systems
(NeurIPS), 2022. 2
[62] Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. 2
[63] Binglu Wang. Yongqiang Zhao, and Xuelong Li. Multiple instance graph learning for weakly supervised remote sensing object detection. IEEE Transactions on Geoscience and Remote Sensing, 60:1-12, 2021.
[64] Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal interven- tion for weakly-supervised semantic segmentation. Advances in Neural information Processing Systems, 33:655-666, 2020. 3
[65] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 876-883 885,, 2017.
[66] Youngmin Oh, Beomjun Kim, and Bumsub Ham. Background-aware pooling and noise-aware loss for weakly-supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 69 13-6922, 2021. 3
[67] Anurag Kumar and Bhiksha Raj. Audio event and scene recognition: A unified approach using strongly and weakly labeled data. 2017 International Joint Conference on Neural Networks (JJCNN, pages 3475-3482, 2016. 3
[68] Anurag Kumar and Bhiksha Raj. Audio event detection using weakly labeled data. In Pre- ceedings of the 24th ACM International Conference on Multimedia, page 1038-1047, 2016. 3
[69] Ankit Shah, Anurag Kumar, Alexander Hauptmann, and Bhiksha Raj. A closer look at weak label learning for audio events. arXiv preprint / 09288, 2018.
[70] John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Dis- criminative embeddings for segmentation and separation. In Proceedings of IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 31-35, 2016. 3
13

---

[71] Prem Seetharaman, Gordon Wichern, Shrikant Venkataramani, and Jonathan Le Roux. Class- conditional embeddings for music source separation. In Proceedings of IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 301-305, 2019. 3
[72] Fatemeh Pishdadian, Gordon Wichern, and Jonathan Le Roux. Finding strength in weakness: Learning to separate sounds with weak supervision. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2386-2333 2019.
[73] Efthymios Tzinis, Scott Wisdom, John R. Hershey, Aren Jansen, and Daniel P. W. Ellis. Improving universal sound separation using sound classification. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 100,
2020. 3
[74] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6399-6408, 2019. 4,5
[75] Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, and Jianmin Jiang. A simple pooling-based design for real-time salient object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3917-3926, 2019. 5
[76] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. 6
[77] Jia Deng. Wei Dong, Richard Socher, Li-Jia. Li, Kai Li, and Li Fei-Fei. ImageNet: A Large- Scale Hicrarchical Image Database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
14


Reviews:
[{"review_id": "XEIYonD9Ze", "reviewer": "Reviewer_CQaV", "summary": "The authors propose a weakly-supervised audio-visual segmentation framework called WS-AVS  that predicts sounding source masks from audio and images without pixel-level ground truth masks. It leverages multi-scale contrastive learning in audio-visual fusion to capture multi-scale alignment, addressing modality uncertainty in weakly supervised segmentation. The refined pseudo masks guide training, enabling generating accurate segmentation masks. Experiments show it outperforms weakly-supervised baselines.", "questions": "As above", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The manuscript presents a solution to the challenging problem of performing segmentation tasks without pixel-level annotations by effectively integrating multi-scale contrastive learning and pseudo mask refinement. It is well written and easy to read through.", "weaknesses": "1. Although these methods have been proven effective independently, their combination in this manuscript does not elevate them to a novel plane but rather presents a blend of established tactics, the novelty aspÅect therefore seems insufficient.\n\n2. While the max-pooled cosine similarity function works well in [9], is there any other similarity function that can be compared with, as AVS has multi-source scenarios?\n\n3. As the WS-AVS framework relies heavily, to some extent, on the generation of pseudo-labels, so is there any other ablations on pseudo-label generation?\n\n4. Noticed that there is no schematic for multi-source in the manuscript, wondering what the pseudo-label for multi-source looks like.\n\n5. In Line 188-190, the authors wrote: the recent audio-visual segmentation baseline [1] does not give any cross-modal constraint on the audio and visual representations in the audio-visual fusion, which causes a significant performance drop when removing the ground-truth masks during training. I notice that [1] proposed an audio-visual matching loss that is used in the multiple sound source case as [1] claimed this loss does not bring further improvement in single-source case. I guess the pixel-level ground truth provides enough supervision in single-source AVS. However, in the studied weakly-supervised AVS setting, will this loss help in the audio-visual fusion?\n\n5. Some minor issues:\n\nTypo. Line 54, “WS-VAS, that” should be “WS-AVS, which”\n\nTypo. Line 176, “V^s” should be “V^s_i”\n\nTypo.  Line 194 “WSSL” should be “WSSS”\n\nThe result in F-score of Multiple Source in Table 1 should be bold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CQaV, \n\nThank you for the detailed review. We will address your concerns below.\n\n> Clarification on novel aspects.\n\nOur paper includes two main novel aspects:\n\n1) We are the first to investigate a new weakly-supervised multi-modal problem that predicts sounding object masks from both audio and image without needing pixel-level annotations.\n\n2) We present a novel framework for weakly-supervised audio-visual segmentation with multi-scale multiple-instance contrastive learning to achieve state-of-the-art results on single-source and multi-source sounding object segmentation.\n\n\n> Other similarity function.\n\nYes. We tried to replace the cosine similarity function with the Kullback–Leibler divergence in multi-source scenarios. The results are shown in the Table below. As can be seen, using the max-pooled cosine similarity function achieves much better performance in terms of both metrics.\n\n| Similarity Function         | mIoU (↑)  | F-score (↑)  |\n| ---- | :----: | :----: | \n| Kullback–Leibler divergence | 24.85     | 29.69        |\n| cosine similarity           | **30.85**     | **46.87**        |\n\n\n> Ablations on multi-scale visual feature maps for pseudo-label generation.\n\nThis is a good suggestion! We abated the stage of multi-scale visual feature maps from {1, 2, 3, 4} for pseudo-label generation, and reported the comparison results on multi-source segmentations in the Table below. When the number of stages is 4, we achieve the best segmentation performance in terms of all metrics. With the increase of feature stages from 1 to 4, the proposed WS-AVS consistently raises results, which shows the importance of multi-scale visual features in pseudo-label generation.\n\n| Feature Stage | mIoU (↑)  | F-score (↑)  |\n| ---- | :----: | :----: | \n| 1             | 26.78     | 31.52        |\n| 2             | 27.19     | 33.56        |\n| 3             | 28.65     | 37.16        |\n| 4             | **30.85**     | **46.87**        |\n\n> Pseudo-label for multi-source scenarios.\n\nFor pseudo-labels for multi-source scenarios, we use the background activation maps as pseudo-labels and categories of each source to further train a salient object [67]. Then generated pseudo-mask separately for each sound source according to Equation (6) by replacing the salient regions with class-specific salient regions $S^\\prime\\in\\mathbb{R}^{M\\times H\\times W}$, where $M$ denotes the number of sound sources. Finally, we apply the loss defined in Equation (7) on each source. We will add this clarification to the revision.\n\n> Audio-visual matching loss in the weakly-supervised AVS.\n\nThis is a good suggestion! We tried to add the audio-visual matching loss to both baseline and our WS-AVS, and reported the comparison results on multi-source segmentations under weakly-supervised AVS setting in the Table below. Applying this loss benefits the audio-visual fusion for weakly-supervised AVS in terms of all metrics. \n\n\n| Method        | AVM     | mIoU (↑)  | F-score (↑)  |\n| ---- | :----: | :----: | :----: | \n| AVS           | &cross; | 8.76      | 15.72        |\n| AVS           | &check; | **9.85**     | **17.36**        |\n| WS-AVS (ours) | &cross; | 30.85     | 46.87        |\n| WS-AVS (ours) | &check; | **32.03**     | **49.15**        |\n\n\n> Minor issues.\n\nThanks for spotting these. We have fixed them. \n\n"}, {"review_id": "2XV9vmiOcj", "reviewer": "Reviewer_yiGS", "summary": "This paper proposes a new contrastive learning approach with pseudo-mask generation/refinement process for weakly supervised video segmentation with audio guidance. The authors show that their method outperforms several other state-of-the-art methods for the visual segmentation task when training and testing on the AVSBench dataset. \n", "questions": "Have you considered an analysis of the proposed approach for the choice of the model for ensuring robustness towards the “reliable pseudo mask extraction with multi-scale visual features”?\n", "limitations": "The authors discuss some limitations of their work and I think they should include the issues that I raised in the Section above and that they will not be able to address through experiments and/or rebuttal. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper shows strong results compared to other methods in the literature.\n- The paper tries to tackle a very important problem in the vicinity of audio-visual perception which is how to find the pixel regions that correspond to sounding sources when limited annotated data are available.\n- The paper is well written and the ideas are clearly presented.\n", "weaknesses": "Unfortunately the paper also contains several weaknesses that narrow down the scope and the potential impact of the paper that could be addressed using experiments and rewriting some parts of the manuscript when needed.\n\n- The authors only include experiments with a very small scale in terms of multiple aspects (video frames, classes of sounds only 23 objects, number of videos < 5k) to compare their method with other established methods. Although having smaller scale experiments help with finetuning of the hyperparameters and early experimentation, it becomes difficult to extrapolate the findings of this study to large scale solutions and real-world data with only showing results on the chosen dataset. Some dataset that come on top of my head and could be used by the authors to enrich the results and show the true potential of their method would be some more limited datasets like Flickr-SoundNet [A], a medium-sized mostly single-source dataset like VGG Sound Source [B] or a real-world video dataset like AudioSet [C] or YFCC100M [D].\n- Building upon my previous argument, using more realistic and large scale datasets would not only put away potential skepticism revolving around the validity of training neural networks with only 5k examples and limited video resolutions (e.g. capturing motion features when the average video has 2 frames is almost impossible for the dataset used in this paper) but would also show whether the proposed approach is robust when trained with videos that contain no on-screen objects which is very difficult to solve for related problems like on-screen sound separation [E]. I don’t want to extend the problem to cases where more than one sounds appear on-screen / off-screen and thus some audio separation front-end might be needed before obtaining the audio features but I would still expect some more convincing large-scale experiments.\n- In the current version of the paper, the authors only consider evaluation datasets where the sounding object appears on-screen at some point in the video but they completely omit to report the performance for off-screen-only videos. To that end, the authors should also evaluate the performance of their model when there is some visual activity on-screen but the sounding object/action cannot be seen in camera and test how robust their model is. According to my experience, there will be a trade-off in the sensitivity of how accurate the model will predict the masks for on-screen objects which correspond to the on-screen audio and how well the model will predict zeros for on-screen objects and actions that do not make any sound (e.g. a human talking in the background while a non-talking human is displayed on-screen).\n\nOverall, I am willing to increase my score if most of the most important above concerns are addressed by the authors since I truly believe that the paper has a great potential to become a good benchmark method for weakly supervised audio-guided video segmentation.\n\n[A] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018.\n\n[B] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020.\n\n[C] Gemmeke, J.F., Ellis, D.P., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M. and Ritter, M., 2017, March. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 776-780). IEEE.\n\n[D] Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D. and Li, L.J., 2016. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2), pp.64-73.\n\n[E] Tzinis E, Wisdom S, Jansen A, Hershey S, Remez T, Ellis D, Hershey JR. Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds. In International Conference on Learning Representations 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer yiGS, \n\nThank you for the detailed review. We will address your concerns below.\n\n\n> Large-scale experiments.\n\nThis is a great suggestion! We extended our method to experiments on Flickr-SoundNet and VGG Sound Source, and reported the comparison results of CIoU with existing approaches in the Table below. Compared to previous methods, our WS-AVS achieves the best results in both benchmarks.\n\n| Method        | Flickr-SoundNet | VGG Sound Source |\n| ---- | :----: | :----: |\n| DSOL [a]      | 74.00           | 29.91            |\n| EZVSL [b]     | 83.94           | 38.85            |\n| SLAVC [c]     | 86.40           | 39.80            |\n| WS-AVS (ours) | **91.63**           | **42.72**            |\n\n> Robustness to non-sounding and off-screen objects.\n\nThis is a good suggestion! To verify the robustness of our proposed approach to non-sounding and off-screen objects, we evaluated our WS-AVS on Extended Flickr-SoundNet and  Extended VGG-SS proposed in [c]. The comparison results of max-F1 with previous methods are shown in the Table below. Compared to the state-of-the-art frameworks, our WS-AVS achieves the best performance in both datasets.\n\n| Method        | Extended Flickr-SoundNet | Extended VGG Sound Source |\n| ---- | :----: | :----: |\n| DSOL [a]      | 49.40                    | 25.60                     |\n| EZVSL [b]     | 54.60                    | 30.90                     |\n| SLAVC [c]     | 60.10                    | 41.50                     |\n| WS-AVS (ours) | **65.27**                    | **52.38**                     |\n\n\n> Robustness to off-screen objects.\n\nPlease see the response above.\n\n\n> Analysis of the choice of multi-scale visual feature maps for pseudo mask extraction.\n\nYes. This is a good question! We abated the stage of multi-scale visual feature maps from {1, 2, 3, 4} for pseudo-label generation, and reported the comparison results on multi-source segmentations in the Table below. When the number of stages is 4, we achieve the best segmentation performance in terms of all metrics. With the increase of feature stages from 1 to 4, the proposed WS-AVS consistently raises results, which shows the importance of multi-scale visual features in pseudo-label generation.\n\n| Feature Stage | mIoU (↑)  | F-score (↑)  |\n| ---- | :----: | :----: | \n| 1             | 26.78     | 31.52        |\n| 2             | 27.19     | 33.56        |\n| 3             | 28.65     | 37.16        |\n| 4             | **30.85**     | **46.87**        |\n\n\n**References**\n\n[a] Hu, et al. Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching. In NeurIPS, 2020.\n\n[b] Mo, et al. Localizing Visual Sounds the Easy Way. In ECCV, 2022.\n\n[c] Mo, et al. A Closer Look at Weakly-Supervised Audio-Visual Source Localization. In NeurIPS, 2022.\n\n\n\n"}, {"review_id": "KUyRyQcHU8", "reviewer": "Reviewer_DJ6f", "summary": "This work introduces a new setting for audio-visual segmentation, Weakly-Supervised Audio-Visual Segmentation (WS-AVS). The authors address the challenge of the costly and not always available pixel-level masks by employing weakly-supervised audio-visual segmentation. This framework uses multi-scale multiple-instance contrastive learning for  capturing multi-scale audio-visual alignment. Tested on the AVS Bench, WS-AVS demonstrates superior performance in both single-source and multi-source scenarios when compared to previous methods. \n", "questions": "The reviewer believes the task of weakly-supervised audio-visual segmentation is worth exploring, and hence currently vote for borderline accept. However, the authors should adequately discuss the key novelty of this paper for the final acceptance.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. To eliminate the requirement of pixel-level annotations, this work propose to solve the audio-visual segmentation problem in a weakly-supervised way. Correspondingly, it uses adopts Audio-Visual Fusion with Multi-scale Multiple-Instance Contrastive Learning and Pseudo Mask Refinement by Contrastive Class-agnostic Maps to solve the challenges in the weakly supervised setting.\n\n\n2. The authors provide comprehensive evaluation under various setting to illustrate the performance of the proposed method.", "weaknesses": "1. Clarifying Novelty\nThe paper's novelty looks somewhat unclear. This reviewer recommends clearly defining the unique technical contributions proposed by this work, especially those set apart from the developments in weakly-supervised segmentation. For instance, it would be advantageous to underscore the importance of Multi-scale Multiple-Instance Contrastive Learning in the given task and its distinctiveness from similar concepts previously employed in weakly-supervised segmentation techniques.\n\n2. Further analysis\nThe authors offer sufficient ablation studies to ascertain the performance improvement attributed to AVF and PMR. However, the reviewer anticipates a more in-depth analysis of the outcomes. Specifically, what distinguishes the audio-visual relationship as learned via supervised methods from those learned in weakly-supervised ways? Considering the final classification outcome, it's intuitive to hypothesize that weakly-supervised methods might result in a sub-optimal or noisy audio-visual mapping. But, without human bias, these weakly-supervised maps might demonstrate superior generalization or aid in identifying human annotation-induced preconceptions. The reviewer encourages exploring these insights rather than viewing weakly-supervised audio-visual segmentation merely as a workaround for data limitations.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DJ6f, \n\nThank you for appreciating our approach. We address your comments below.\n\n\n> Clarification on contributions/novelty.\n\nOur paper includes two main technical contributions:\n\n1) We are the first to investigate a new weakly-supervised multi-modal problem that predicts sounding object masks from both audio and image without needing pixel-level annotations.\n\n2) We present a simple but effective framework for weakly-supervised audio-visual segmentation by combining multi-scale multiple-instance contrastive learning and pseudo mask refinement to achieve state-of-the-art results on single-source and multi-source sounding object segmentation. \n\nBased on the new problem and effective method, we believe that our paper has great potential to become a good benchmark approach for weakly supervised audio-visual segmentation.\n\n\n\n> Further analysis on audio-visual matching in supervised/weakly-supervised AVS.\n\nThis is a good suggestion! To explore the effect of audio-visual matching loss introduced in supervised AVS [1] for multi-source segmentations, we added the audio-visual matching loss to both the baseline and our WS-AVS, and reported the comparison results on multi-source segmentations under weakly-supervised AVS setting in the Table below. Applying this audio-visual matching loss benefits the audio-visual fusion for weakly-supervised AVS in terms of all metrics. Besides, the magnitude of improvement is large than 1% reported in supervised AVS [1], which indicates that the audio-visual relationship learned in weakly-supervised AVS is more important than that learned from supervised settings.\n\n\n| Method        | AVM     | mIoU (↑)  | F-score (↑)  |\n| ---- | :----: | :----: | :----: | \n| AVS           | &cross; | 8.76      | 15.72        |\n| AVS           | &check; | **9.85**     | **17.36**        |\n| WS-AVS (ours) | &cross; | 30.85     | 46.87        |\n| WS-AVS (ours) | &check; | **32.03**     | **49.15**        |\n\n\n"}, {"review_id": "CdQ741HFjP", "reviewer": "Reviewer_XL4i", "summary": "This paper presents a new framework for weakly supervised audio-visual segmentation which does not need pixel level annotations. This is achieved via the proposed multi-scale multiple-instance contrastive learning approach which can capture audio-visual alignment in multiple scales. Comparison with existing methods show that the proposed methodology leads to state-of-the-art results.", "questions": "Please see the questions above.", "limitations": "As mentioned above, it would be desirable if the authors include additional failure cases in the supplementary material. At the moment there is only one.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes an interesting approach ti audio-visual segmentation which does not rely on pixel-level annotations. This is an important contribution since obtaining such annotations is time consuming.\n\nThe paper is easy to follow. \n\nConvincing results are presented and a detailed ablation study is presented.", "weaknesses": "The main weakness of the paper is that several important details are not described.\n\n- In table 1, results for multiple sources are presented without providing details, e.g., how many sources are used, how were these examples obtained? \n\n- It is not clear how the multi-scale features are generated.\n\n- How are the negative examples selected in multi-scale multiple instance contrastive learning? Would be very helpful if the authors elaborate on this.\n\n- Equal weights are used for the two loss terms in Eq. 8. Have the authors considered using different weights? This might improve the performance.\n\n- It is not clear why the face and body are predicted as the sound producing regions in case a person speaks. Ideally, just the face or just the mouth should be identified as the sound source.\n\n- The audio which corresponds to the images in Fig.2 is not presented. Most of the examples are included in the supplementary material so adding such a comment in the caption would be helpful.\n\n- Finally, it would be helpful to add additional failure cases. Only one example is included in the supplementary material. \n\nAnother weakness, is that only one dataset is used for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer XL4i, \n\nThank you for appreciating our approach. We address your comments below.\n\n\n> Details on multiple sources.\n\nWe use two or three sound sources from 2,120 frames for evaluation. For pseudo-labels for multi-source scenarios, we use the background activation maps as pseudo-labels and categories of each source to further train a salient object [67]. Then generated pseudo-mask separately for each sound source according to Equation (6) by replacing the salient regions with class-specific salient regions $S^\\prime\\in\\mathbb{R}^{M\\times H\\times W}. Finally, we apply the loss defined in Equation (7) on each source. We will add this clarification to the revision.\n\n\n> Clarification on multi-scale features.\n\nThe multi-scale features with sizes of {56x56, 28x28, 14x14, 7x7} are generated from four stages ([3,4,6,3]) of the ResNet50-based image encoder.\n\n\n> Negative examples.\n\nFollowing the spirit of contrastive learning methods, negative examples in multi-scale multiple instance contrastive learning are obtained from other images in the same mini-batch. \n\n\n> Different weights.\n\nYes. We have explored different weights but did not see significant improvements (0.1%) in the performance.\n\n\n> Clarification on the prediction of face and body as the sound producing regions.\n\nThis new weakly-supervised AVS problem is challenging to generate a fine-grained sound source from a small object, such as the mouth. Meanwhile, many objects with diverse semantic might be included in the video frames. Even for the ground-truth, the face and body are also annotated as the sounding sources when a person speaks or sings.\n\n\n> Adding an audio comment.\n\nThanks for the suggestion. We will add the comment to the caption in the updated version.\n\n> More failure cases. \n\nThe model may fail when two sources have very similar visual semantics, e.g., a mixture of people cheering and people crowd. We will add more failure cases to the supplementary. \n\n\n> Evaluation of more datasets.\n\nThis is a great suggestion! We extended our method to experiments on Flickr-SoundNet and VGG Sound Source, and reported the comparison results of CIoU with existing approaches in the Table below. Compared to previous methods, our WS-AVS achieves the best results in both benchmarks.\n\n| Method        | Flickr-SoundNet | VGG Sound Source |\n| ---- | :----: | :----: |\n| DSOL [a]      | 74.00           | 29.91            |\n| EZVSL [b]     | 83.94           | 38.85            |\n| SLAVC [c]     | 86.40           | 39.80            |\n| WS-AVS (ours) | **91.63**           | **42.72**            |\n\n\nTo verify the robustness of our proposed approach to off-screen/silent objects, we evaluated our WS-AVS on Extended Flickr-SoundNet and  Extended VGG-SS proposed in [c]. The comparison results of max-F1 with previous methods are reported in the Table below. Compared to the state-of-the-art frameworks, our WS-AVS achieves the best performance in both datasets.\n\n\n| Method        | Extended Flickr-SoundNet | Extended VGG Sound Source |\n| ---- | :----: | :----: |\n| DSOL [a]      | 49.40                    | 25.60                     |\n| EZVSL [b]     | 54.60                    | 30.90                     |\n| SLAVC [c]     | 60.10                    | 41.50                     |\n| WS-AVS (ours) | **65.27**                    | **52.38**                     |\n\n\n**References**\n\n[a] Hu, et al. Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching. In NeurIPS, 2020.\n\n[b] Mo, et al. Localizing Visual Sounds the Easy Way. In ECCV, 2022.\n\n[c] Mo, et al. A Closer Look at Weakly-Supervised Audio-Visual Source Localization. In NeurIPS, 2022.\n\n\n\n"}]

Author rebuttal:
Dear all reviewers:


We extend our heartfelt gratitude to each of you for generously dedicating your valuable time and expertise to reviewing our work. We acknowledge and deeply appreciate the insightful comments and critiques provided by all the reviewers. In response to your invaluable feedback, we have made significant revisions to our manuscript, aiming to address each of your concerns in a comprehensive and scholarly manner. Reviewer CQaV and Reviewer yiGS, we kindly request your reconsideration of your decision, given that we have taken utmost care to thoroughly address the main comments raised in your reviews. 


Once again, we express our sincere appreciation for your valuable contributions to the review process. Your expertise and guidance have been invaluable in improving the quality of our work. We remain committed to continuous improvement and eagerly await your final decision.


Meta-review: One reviewer recommends rejection, and three recommend acceptance. The reviewer in favor of rejection (Reviewer CQaV) initially raised several concerns. After the rebuttal, they increased their score but still recommended rejection due to what they perceived to be limited novelty. The reviewers in favor of acceptance generally praised the quality of the results (yiGS, XL4i), presentation (yiGS), and the reduction in the required annotation effort (XL4i), and evaluation (DJ6f). On balance, the reviewer weighs the quality of the results over the limited novelty, and recommends acceptance. Authors should attend to main points in the reviews when preparing a final version.
Decision: Accept (poster)