Unsupervised Sounding Pixel Learning
Authors: Yining Zhang
Yanli Ji
Yang Yang
Venue: EMNLP 2023 Main (2023)
Keywords: unsupervised
cross-modal learning
sound source localization
semantic affinity refinement

Abstract: Sounding source localization is a challenging cross-modal task due to the difficulty of cross-modal alignment. 
Although supervised cross-modal methods achieve encouraging performance, heavy manual annotations are expensive and inefficient. Thus it is valuable and meaningful to develop unsupervised solutions. 
In this paper, we propose an **U**nsupervised **S**ounding **P**ixel **L**earning (USPL) approach which enables a pixel-level sounding source localization in unsupervised paradigm. 
We first design a mask augmentation based multi-instance contrastive learning to realize unsupervised cross-modal coarse localization, which aligns audio-visual features to obtain coarse sounding maps. 
Secondly, we present an *Unsupervised Sounding Map Refinement (SMR)* module which employs the visual semantic affinity learning to explore inter-pixel relations of adjacent coordinate features. It contributes to recovering the boundary of coarse sounding maps and obtaining fine sounding maps. 
Finally, a *Sounding Pixel Segmentation (SPS)* module is presented to realize audio-supervised semantic segmentation. 
Extensive experiments are performed on the AVSBench-S4 and VGGSound datasets, exhibiting encouraging results compared with previous SOTA methods.

Unsupervised Sounding Pixel Learning
Yining Zhang', Yanli Ji #1.2 and Yang Yang!
School of Computer Science and Engineering, UESTC
Shenzhen Institute for Advanced Study, UESTC
Institute of Electronic and Information Engineering of UESTC in Guangdong, China zynêstd. uestc. edu. cn, {yanliji, panni,,,yang.... yaanggyangeeuuteeeeo cn.
Abstract
Sounding source localization is challenging cross-modal task due to the difficulty of cross- modal alignment. Although supervised cross- modal methods achieve encouraging perfor- mance, heavy manual annotations are expen- sive and inefficient. Thus is valuable and meaningful to develop unsupervised solutions. this paper, we propose an Unsupervised Sounding Pixel Learning (USPL) approach which enables pixel-level sounding source localization in unsupervised paradigm. We first design mask augmentation based multi- instance contrastive learning to realize unsuper- vised cross-modal coarse localization, which aligns audio-visual features to obtain coarse sounding maps. Secondly, we present an Un- supervised Sounding Map Refinement (SMR) module which employs the visual semantic affinity learning to explore inter-pixel relations of adjacent coordinate features. It contributes recovering the boundary of coarse sound- ing maps and obtaining fine sounding maps. Finally, a Sounding Pixel Segmentation (SPS) module's is presented to realize audio-supervised semantic segmentation. Extensive experiments performed on the AVSBench $4 and VG- GSound datasets, exhibiting encouraging re- sults compared with previous SOTA methods.
Potts
Cron
DEC.
Mame! ansucation
Rafine
Pusion
EVC,
PAG.
AMG
(a) Unsup SSL.
(b) Sup SSL.
(c) Our USPL.
Figure I: Comparison of USPL with previous
SSL tasks. (a) Unsupervised Sound Source Localiza- tion (Unsup SChn et al., 2021) calculates audio- visual cross attention obtain patch-level coarse sults. (b) Supervised Sound Source Localization (Sup SSL)Zhou et al., 2022) provides a supervised label for model training. (c) Our Unsupervised Sounding Pixel Learning (USPL.) aims to obtain pixel-level localization results an unsupervised paradigm.
in a given audio-visual scene without any annota- tions.
Despite recent advancements in SSL, some chal- lenges remain be addressed. One of the pri- mary challenges
learn accurate pixel-level sounding maps rather than approximate patch-level maps. As shown in Fig. 1(). Unsup SSL methods (Chen et al., 2021) always learn the audio-visual co-occurrence by contrastive learning (Chen et al.. 2020b), then directly calculate the audio-visual attention to estimate the sounding maps. How- ever, these approaches often output coarse and unreliable-sounding masks containing only local- ization information but lack object shape informa- tion.
1 Introduction
Audio-visual localization tasks have attracted much attention recent years. Various works have made great achievements, including Sound Source Local- ization (Chnn et al., 2021; Mo and Morgado, 2022b), Audio- Visual Event localization (AVE), Audio-Vissee Video Parsing (AVVP)(Wu and Yang, 2021; Lin et al., 2021), and Audio Visual Segmen- tation (Zhu) al., 2022). In this work, we focus on variant of SSI. task unsupervised pixel- level sound source localization (pixel SSL), which aims at accurately localizing the sounding objects
Another challenge is doing fine- fine-grained sound source localization in totally unsupervised paradigm. As depicted in 1ib. precisely identify regions of sounding, Sup SSL methods (Zhou al., 2022) build a pixel-level labeled dataset and utilize semantic segmentation models to produce pixel-level results. However, Sup SSL

---

methods are limited by the requirement for inten- sive annotations, which can be burdensome task and hard to be feasible in real environments. There- fore, in Unsupervised Sounding Pixel Learning (USPI,, Fig. I(c)), our motivation to produce precise pixel-level sounding object maps in videos by a totally unsupervised paradigm.
We adopt sounding pixel segmentation (SPS) module for lightweight evaluation and facil- itating the SMR converge to better perfor- mance.
The final segmentation results outperform SOTA unsupervised methods by 9.25% in mloU the AVSBench-S4 and by 2.62% in cloU on the VGGSS benchmark.
To address the challenge of pixel SSL, first conduct mask augmentation based multi-instance contrastive learning (MMICL) to align the paired audio-visual features. We highlight that the mask augmentation layer greatly alleviates the overfit and sub- uub-optimization problem proposed in previous work(Chen et al., 2021; Mo and Morgado, 2022b). After completing the MMICL, can use audio- visual attention obtain patch-level sounding ob- ject results which are considered as coarse sound- ing maps.
2 Related Work
2.1
Sounding source localization
Sound Source Localization (SSL) aims to locate the sounding regions in visual frames. et al.. 2022) Recent approaches (Arandjelovic and Zisser- man, 2018: Senocak et al., 2018; Hu et al., 2019; Afouras et al., 2020; Hu et al., 2020; Liu et al., 2022a; Zhou al., 2022; Song et al., 2022; Liu et al., 2022b; Sun et al., 2023) extensively lever- aged contrastive learning based on audio-visual Co-occurrence to address this issue unsupervised as shown in Fig. (a). For example, (Arandjelovic and Zisserman, 2018) formulated the problem in Multiple Instance Learning (MIL) framework and trained localization network in the manner of Audio Visual (aaappoonndnnerraanniiileele Zisserman, 2017). Accordingly, (Chen et al., 2021) incorporated explicitly background regions with low correlation to the given audio into the frame- work and regarded them as hard negative in con- trastive learning. (Sun et al., 2023) investigated the false negative issue in audio-visual contrastive learning and proposed a method to alleviate this problem. In (Qian et al., 2020), the authors ex- plored for multi-source localization.
However, these coarse sounding maps fall short of our pixel SSL requirements. Therefore, we per- form an unsupervised Sounding Map Refinement (SMR) module obtain fine sounding maps con- taining shape information. SMR predicts seman- tie affinities of paired adjacent coordinates in the image by mnter-pixel relation mining. Then, SMR produces fine sounding maps by revising the coarse sounding maps according to the affinity matrix. Af- ter these processes, SMR significantly refines the coarse sounding maps, enabling object boundary information to be recovered.
and
Although SMR produces fine sounding maps, it is too heavy for evaluation. Thus we propose Sounding Pixel Segmentation (SPS) module. SPS is an encoder-decoder architecture that includes an audio signal supervised. We use the fine sounding maps generated by SMR to train the SPS. We also find that SPS can improve the training of SMR to obtain better convergence. Therefore, SPS makes our methods more stable and accurate in sounding map generation.
However, these unsupervised SSL methods only produce inaccurate patch-level results and our goal is to produce fine-grained pixel-level sounding re- sults. The work most closely related to ours is AVS (Zhou et al., 2022), which employed a seg- mentation network trained with label supervision to produce pixel-level sounding maps, as shown in Figure 1(b). However, this work is difficult to apply real scenarios, because it needs burden- intensive human annotations. Therefore, we focus on unsupervised pixel SSL which aims to produce pixel-level localization results for sounding objects in videos without any annotations.
Our major contributions are summarized below:
We design a novel mask augmentation based -nlti-instance contrastive learning that allevi- ates the overfit and aub-optimization problem to realize audio-visual alignment better.
We propose unsupervised sounding map refinement (SMR) module that leverages the visual semantic affinity to recover the sound- ing object boundary and produce fine-grained sounding maps.
2.2 Weakly-supervised semantic segmentation The approuches of weakly supervised semantic segmentation usually start training a classification

---

network with image-level labels and produce the initial pseudo labels using CAM. To address the drawback of incomplete object activation CAM, these methods always build a refined process. (Ahn and Kwak, 2018) involved modeling the pixel-level affinity distance from initial CAMs and employed a random walk to propagate individual class la- bels at the pixel level to refine the result. The random walknaeraaza and Chandraker, 2017) is an operation propagating the sparse labels to pro- duce guessed dense labels which are often used in applying affinity matrix to refine map, (Ru et al., 2022) built a transformer encoder and learned the semantic affinities from the multi-head self- attention. Then it leveraged the learned affinity to refine the initial pseudo labels for segmenta- tion. These methods usually adopt pixel-level re- finement algorithms e.g. dCRI'(Käähenühhl and Koltun, 201 1), PRM(Araslanov and Roth, 2020), PAMR (Ru et al., 2022) to help refining. In this work, we build a Sounding Map (efinement(SMRR) module to refine the coarse sounding map which is similar to (Ahn and Kwak, 2018).
and F are features after projectors mapping. We compute the audio-visual cross attention obtain the coarse sounding maps Mers in Eq. 1.
{Fa, Fe) || Fall F
Mers
(1)
Mask augmentation based Multi-Instance Con- trastive Learning (MMICL) Following the previ- ous EZ-VSL (Mo and Morgado, 2022b), we con- duct multiple-instance contrastive learning (MICL) to align the audio-visual features. Specifically, we define the visual features from various loca- tions in the paired audio-visual set positive bag pi (Ft) while those from unpaired sets nega- tive bags No {Fr t I....N, o i}. Obviously, the audio feature is matched with sounding posi- tions visual feature in the positive bag Ps while mismatched at all visual features from negative bags N
Following the mentioned MICL, we find that the model is prone to overfit and often locates part of sounding objects. Most likely because the MICL makes an inappropriate assumption that the audio feature is matched at only one visual location in the positive bag. This assumption makes the model easy to overfit in focusing on the most matched visual regions and failing to locate all sounding positions. To combat overfitting and mine the complete sounding objects, propose Mask aug- mentation based Multi- IantanIesttaeee Contrastive Learn- ing(MMICL) when training. contains a random mask augmentation layer which randomly drops the visual features of some locations during training to ensure potential sounding locations have more opportunities for optimization. The MMICL loss is defined in Eq. 2.
3 Methodology
In this section, we introduce the proposed method Unsupervised Sounding Pixel Learning (USPL) to conduct the pixel SSL. task. In USPL, we first align the audio and visual features in Section 3.1 obtain the coarse sounding maps. Then, propose refined module named Sounding Map Refinement (SMR) to recover the boundary from the coarse sounding maps in Section 3.2. Next, we apply the fine sounding maps generated by SMR to train the Sounding Pixel Segmentation (SPS) module in Section 3.3.
3.1 Unsupervised cross- modality coarse localization
exp( cop(=sim( Fi. ((F))) Commiee Σ log
E, expppssmmmmm (r. k!))
Given an audio-visual dataset D {D., ii= 1,..., NN. we extract two modality features by en- coders f.). fo()), respectively. The audio en- coder, here we use a pretrained VGGish, extracts global audio features FA - fa:). Fi E RC. The visual encoder, here we use pretrained ResNet50, produces set of features spanning all locations in image as F = (v:), Ft I Txxxxx Then, we utilize two projectors (-) and (.).). one for each modality feature, to map the features into shared feature space. The projection process can be real- ized by F = 9a (F)), F = g. (..F),,, where the Fri
where is the temperature parameter, and the sim- ilarity sim(Fa F?)) between an audio feature F. and bag of visual features Fi is computed by max-poolia max-pooling audio-visual cross attention.
3.2 Unsupervised Sounding Map Refinement
In order localize sounding objects with high precision, we build Sounding Map Refinement (SMR) module to recover the object shape informa- tion from the coarse sounding maps M

---

PMV- aff
F
Inter-pixel relation mining
Refine using
Laff
SMR
Aftiaity A
Affinity label Yat
Fine sounding map M
ResNet50
Fv
Frame
Enendor
Lseg
Random mask
sounding
nugmentation layer
A Audlo-visual
map Mers
Visual Emender
Visual Decoder
Lumiiet
VGGlsh
Fu
ResNet50
F
FPN
Sound
Audio Encoder
Frame
dec
Fg.
SPS
The proposed Unsupervised Sounding Pixel Learning (USPL) method. We first extract paired audio- visual feature Fa. F, using each modality extractor. Then, we propose a mask augmentation based multi-instance contrastive learning which aligns the audio-visual feature to generate coarse sounding maps Mas. In the Sounding Map Refinement (SMR) module, we derive the affinity matrix A from the affinity feature Fatt by mining the inter-pixel relation in the image. Next, employ the learned affinity revise the Mers and output fine sounding maps. The Sounding Pixel Segmentation (SPS) module produces the final segmentation sounding map results. The Process of Multi-scale Visual feature (PMV) block contains two process streams to provide multi-scale feature for SMR and SPS which is detailed in Fig.3
Figure 2:
Specifically, SMR builds an affinity matrix to predict the semantic affinitics between adjacent coordinate features in an image. This involves ana- lyzing the relationship between different locations of image features. Once affinity matrix A has been predicted, SMR uses A as transition probabilities revise coarse sounding map.
how to generate affinity labels and how to train the SMR with affinity labels.
For the purpose of training SMR to generate the affinity matrix A, we first obtain semantic affinity labels Yaf E how from the coarse sounding map Mass Our approach involves identifying fident areas of the sounding object and background from the coarse sounding map and then sampling training examples from these regions. In order obtain better alignment with the low-level image appearance, we use pixel-level refinement algo- rithm Pixel-wise Adaptive Mean Shift (PAMR) (Ru et al., 2022) to refine the coarse sounding maps in Eq. 4.
Semantic affinity. As shown Fig.2, SMR first computes the affinity matrix from the affin- ity feature Fatt The affinity feature is produced by the PMV-aff block which is detailed in Fig. 3. To generate the affinity feature Fatt, we concate- nate the multi-scale visual features extracted from ResNet50 and then upply MLP layer to aggre- gate these hierarchical features. After extracting the affinity feature, we compute the Affinity matrix by inter-pixel relation mining. Specifically, we identify a set of locations that are within a distance of in coordinate space. The semantic affinity be- tween neighborhood features i and in this set is denoted by A(i,j) defined in Eq. 3.
M,f PAMR (Mers)
(4)
Then, we employ two background thresholds B and 3h where 0 < 3 Br < 1, to generate a Tri-map Yin which delineates the reliable sounding area, silent area, and uncertain area as defined in Eq. 5.
1,
if M, 6h
A(i.)) - erp { Fan (24, v1) Fan Far 85 1b
Yui v)
0, if M, < B
(5)
(3)
255, otherwise
The SMR module needs semantic affinity labels for paired features confirm their relations. However, because of the unlabeled setting, no affinity labels can be obtained directly. In such case, we generate affinity labels from the coarse sounding map Mass instead. the following paragraphs, we present
where and denote the sounding area and the silent area, and 255 is the ignore area. Yai v) is the tri-map coordinate (x, y).
The pscudo affinity label Yuu is then derived from Yin Specifically, in tri-map You. the (z,, y:

---

affinity and suppresses the wrongly activated gions so that the activated maps align better with the semantic boundaries.
Fail
MLP
SMR
Stage1
Stage2
Stage3
Stage4
After the refinement, the fine sounding maps Mine are fine- tree-gaaied enough for training sounding pixel segmentation networks.
PhRtoatf
PHV--bc
FPN Decoder
F..
Frame
ASPP
3.3 Sounding Pixel Segmentation (SPS)
Visual Enooder
Although the SMR produces fine-grained sounding maps, it is too heavy for evaluation. Thus, we adopt the Sounding Pixel Segmentation (SPS) module shown in Fig.2. We also observe that SPS im- prove the training performance of SMR produce more accurate localization results.
The Process of Multi-scale Visual feature (PMV). To leverage the multi-scale visual features in SMR and SPS, we extract multi-stage features from ResNe150. Subsequently, two post-processing blocks, PMV-aff and PMV-dec, the affinity feature Fan and residual feature set {Fi. F2, 33 F4), which are then passed to subsequent modules. @ means concatenate.
Figure 3:
constructed to produce
Audio-visual interactive Encoder. As illus- trated in Fig.2, the visual encoder of SPS is struc- turally equivalent to the encoder employed MMICL, and these two encoders share weights. Therefore, we use the visual feature extracted in Sec. 3.1 for SPS. Here we use the coarse sound- ing map Mers fuse the audio-visual features defined in Eq. 9.
and (zj, Mj) are from the same semanttc area, we set their affinity labels as positive Y otherwise, their affinity labels are set negative - If either (ms, yt) (xy, yj) is sampled from the ignored area, their affinity labels will be set as ignored. Once the affinity label Yur has been obtained, we can train the SMR under supervision to predict the semantic affinity matrix A as defined in Eq.
Fas = F. + Fg O norm(Mard)
(9)
where Fav is the audio visual fusion feature, norm is Min-Max Normalization. O is Hadamard prod- uct.
(6) Call = -Y+ . log(A))))))))))) - art log(1 A) uit
As shown in Fig.3, after getting multi-scale visual features from various stages, we use Atrous Spatial Pyramid Pooling (ASPP) (Chen et al., 2017) to produce a residual feature set [F1, F2, Fa. F4] for subsequent processing. Segmentation decoder. We here use the FPN- decoder(Kirillov et al., 2019) in this work for its flexibility and effectiveness. The audio-visual fu- sion feature and residual feature set are fed into the decoder. In short, the j-th decoder layer, both the feature from residual feature set F5 and decoder stage Fd are utilized for the decoding process. The decoded features are then upsampled to the next stage. The final output of the decoder is M E RTxH>W activated by sigmoid.
where y is the positive label in Yaf and a is the negative one.
Sounding map refinement. The learned reli- able semantic affinities between paired adjacent coordinate features could be utilized to refine the coarse sounding maps. This process is achieved through random walk, as demonstrated in (Vernaza and Chandraker, 2017). For the learned semantic affinity matrix A, the semantic transition matrix T is defined by Eq. 7.
T D A whereDi Σ>
A
(7)
where 3 is hyper-parameter ignore trivial affin- ity values in A, and is diagonal matrix to nor- malize A. The propagation of refinement for the coarse sounding map Mass is defined by Eq.
Given the prediction M and the SMR fine sound- ing map Mine, we straightforwardly utilize the bi- nary cross-entropy loss, similar supervised train- ing for network training which is defined in Eq. 10.
Mine T x vec( Mers)
(8)
Leeg = - Mane (Mg
(10)
where vee(.) means vectorization of a matrix, Mine is the fine sounding map. This propagation process diffuses the semantic regions with high
3.4 Loss settings
As shown in Fig.2, our framework comprises three loss terms, a mask augmentation based multi-

---

Methods
Sup.
Modality
mloU (%)
F-score
3DC(Mahadevan al., 2020)
F
V
57.10
0.759
iGAN(Mao et al., 2021)
F
V
61.59
0.801
AVS (Zhou et al., 2022)
JF
A+V
78.74
0.879
LAVISH(Lin et al., 2023)
F
A+V
80.10
0.891
CAM(Zhou et al., 2016)
I
V
20.40
0.387
IRNet(Ahrr et al., 2019)
I
V
35.68
0,475
MSSL(Qian et al., 2020)
I
A+V
44.89
0.663
FNAC(Sun al., 2023)
U
A+V
32.46
0.434
EZ- Voooo and Morgado, 2022b)
U
A+V
34.01
0.562
LVS (Chen al., 2021)
4
A+V
37.94
0.510
USPL(Ours)
U
A+V
47.19
0.617
Table 1: Performance on AVSBench-S4 test, compared to several related methods. Sup. denotes supervision types. F: train under full supervision; train under weakly supervision with image-level label; U: train under unsupervision with no label. Modality denotes the modalities of imput data. V refers to visual only; A+ refers to audio and visual.
instance contrastive learning loss Lmmicl, refine training loss Laff, and segmentation loss Leeg- Since annotations exist in method, the model heavily relies the information obtained from cor- respondence audio-visual data, we first only use mmmisit training the model for 3 epochs make our model more stably. After that, we use the overall loss defined in Eq. to train the model. The overall loss is weighted sum of Emmicl, Laffs Loog-
The images are resized to 224 x 224 resolution. The audio inputs of the AVSBench-S4 dataset are is clips and the audio inputs of VGGSound are 3s clips that contain the selected frame. Here the raw audio waveform has been converted 257 x 301 log-mel spectrogram by Short-Time Fourier Transform (Ayta)(Aytar et al., 2016). We adopt a VGGiis((Hershe et al., 2017) pretrained on Audio-Set as audio encoder and slightly modified ResNer50(He et al., 2016) pretrained ImageNet as visual encoder, respectively, We modify the last conv-stride of Resnet50. Besides, remove the last linear layer and pooling layer of Resnet50. As shown in Fig.3, the dim of affinity fea- ture extracted by PMV-aff is 256, the dim multi-scale residual feature set [Fi, F2, F3. Fa] are [256, 512, 1024, 2018). The dropout probability of random mask augmentation layer is set to 0.5. We set the cared neighborhood radius 7. We set the background scores B 0.4 and 3, 0.55 in Eq. respectively. The resolution of affinity matrix A in our experiment is 56 x 56. We set 10 in Eq.7 and = 8 in Eq.8. The temperature parameter 0.07. The balanced parameters in overall loss Eq. are A = 0.1, X2 0.4, x = 0.5. The batch size is set as 32. All models are trained with the Adam optimizer using learning rate of 10-4 For training stability, we warm up by MMICL loss for epochs, then we train the model using the overall loss for the remaining 20 epochs.
C = Crmice + X2Caff + >zEseg
(11)
where Åι X2, and λ3 serve weighting factors that balance the contributions of various loss functions.
4 Experiments
4.1 Dateset and Implementation Details
Dataset. We evaluate our approach on two datasets: AVSBench-S4(2ouu et al., 2022) and VGGSound (Chen et al., 2020a). AVSBench-S4 is the single source subset of the AVSBench dataset which con- tains 4,932 vidcos of 5s over 23 categories. VG- GSound dataset is large video dataset containing over 200K videos from 309 categories.
Localization performances are measured on two benchmarks, SVSBench-S4 test and VG- GSound Source (VGGSS)(Chen et al., 2021). The AVSBench-S4 test has 3700 audio-visual pairs with manually pixel-level sounding object labels. VG- GSS has 5,000 audio-visual pairs with manual boundary box-level sounding object labels.
4.2 Comparison with SOTAs
Implementation Details. Our visual inputs are the first frame of each clip in AVSBench-Se S4 and the center frame of videos in VGGSound.
We use two common metries of semantic segmen- tation, Mean Intersection over Union (mIoU) and

---

Methods
Sup.
Modality
cloU-0.5 (%)
AUC (%)
Attention A (SeoSaakk al., 2018)
U
A+V
18.50
30.20
DMC(Hu al., 2019)
u
A+V
29.10
34.80
AVObject(Afoural al., 2020)
u
A+V
29.70
35.70
LVS(Chen et al., 2021)
A+V
34.40
38.20
SSPL(Song et al., 2022)
u
A+V
33.90
38.00
EZ-VSL(Mo and Morgado, 2022b)
U
A+V
34.38
37.70
EZ-VSL(Mo and Morgado, 2022b)
u+0 O
A+V
38.85
39.54
SLAVC(Mo and Morgado, 2022a)
U
A+V
37.22
38.60
SLAVC(Mo and Morgado, 2022a)
u+0
A+V
39.67
39.11
FNAC(Sun al., 2023)
u
A+V
39.50
39.66
FNAC(Sun al., 2023)
u+0
A+V
41.85
40.80
USPL(Ours)
u
A+V
44.47
45.11
Table Performance on VGGSS dataset, compared to various SSL methods. Sup. denotes supervision types. train under unsupervised with no label: U O: unsupervised training but evaluate with the help of a trick OGL(Mo and Morgado, 2022b); Modality denotes the modalities of input data. A+V refers to audio and visual.
F-score to evaluate in the AVSBench-SS S4 test. Addi- tionally, use another two metrics to evaluate in VGGSS, Consensus Intersection over Union(cloU) (Senocak et al., 2018) and Area Under Curve (AUC). For metrics, high values mean better localization performances.
4.3 Ablation Study and Analysis
Module ablation. Our USPL contains three main modules as shown in Fig.2. To explore the impact of each module, we ablate each component indi- vidually in Table 3. All results are obtained follow- ing the same parameter setting AVSBench-S4 val and test set. The first row our baseline EZ- VSL, which gets 33.8 33.85% mloU on the validation set and 34.01% mloU on the test set. The see- ond row the result of using augmentation based multi-instanee contrastive learning which achieves 0.24% promotion on the validation set than base- line. The third row is the result of adding SMR block which boosts performance by visual affinity refinement, achieving a 9.63% mloU promotion on validation set. The fourth row is the result of ablating SMR block while using MMICL and SPS block, which drops 10.08% mIoU comparing with adding SMR. The last row is the result adding the SPS block which gets a more accurate result, achieving 45.41% mloU on the validation set.
AVSBench-S4. As shown in Table 1, we present the comparisons between various approaches on the AVSBench-S4. Our proposed method USPL gets 47.19% mIoU and 0.617 F-score in the AVSBench- $4. It shows that our USPL clearly outperforms pre- vious SOTA unsupervised methods LVS by 9.25% mloU and 0.11 F-score respectively, while even competing with some weakly supervised methods, such as CAM, and MSSL.. is worth mentioning that our unsupervised method achieves remark- able mloU score of 47.19%. which reaches 58.91 % of the best-supervised counterpart LAVISH(Lin al., 2023) AVSBench-S4 test.
VGGSS. As shown in Table 2. Our method also beats all previous SSL methods with a sig- nificant improvement. For example, on the VG- GSS benchmark, we outperform previous SOTA FNAC by 2.62% cloU and 1.31% AUC. It's no- ticed that some methods use abject-guided local- ization (OGL)(Mo and Morgado, 2022b) trick to improve the performance. This trick introduces the visual prior knowledge from the pretrained dataset help localization. We consider that the OGL vio- lates the assumption of unsupervision. Therefore, we don't adopt the OGL in our method and we still get the best performance.
As discussed in Section 3.3, the proposed SPS module can help the convergence of SMR to reach batter performance. We here conduct an experi- ment to prove this. As shown in Table 4, when training SMR alone, the mloU of the fine sound- ing map produced by SMR in AVSBench-SA test 44.01 %, when we train the SMR together with SPS, the mloU the fine sounding map produced by SMR has increased to 46.07%.
Analysis of module efficiency. Here we dis- cuss the model computation and memory costs as shown Table 3. In our proposed USPL, SMR

---

modules
mloU(%))
results
MMICL
SMR
SPS
val
test
Params(M))
FLOPS(G))
FPST
EZ-VSL (Baseline)
33.85
34.01
24.14
6.46
15.45
34.09
35,52
24.14
6.46
15.45
V
43.72
44.01
69.64
40.85
4.34
USPL (Ours)
35.49
35.61
68.12
38.61
9.94
V
45.41
47.19
68.12
38.61
9.94
Table 3: Ablation study of MMICL, SMR, and SPS module of the proposed USPL evaluated on the val and test set of AVSBench-S4 in mloU(%). Analysis computation and memory cost baseline and USPL's modules inference procedure.
mloU%
features dropping during training. As shown Table 5, we perform a series of settings and com- pare the results to the case where no random mask augmentation is used, i.e., ó = 0. It is observed that using random mask always yields better per- formance. Moreover, when achieves its optimal performance. is also impor- tant to note that excessive drop probability can be detrimental to the model's performance.
Only SMR SMR & SPS
44.01
46.07
Table 4: Ablation study of SMR module. We investigate the effects of the SPS module the SMR module.
0.5, the model
and SPS modules play crucial roles. SMR mod- ule refines sounding maps by leveraging visual semantic affinity, resulting in fine-grained sound- ing maps. However, it is noted that SMR exhibits low efficiency. Therefore, we propose SPS mod- ule serving as lightweight module for evaluation. As shown in Table 3, we calculate the parameters (Params), floating point operations (FLOPS), and frames per second (FPS) evaluate the methods' efficiency. We find that using SPS bypass SMR could not only reduce space complexity and sig- nificantly decrease time complexity, but also lead to better performance. However, compared to the baseline, our USPL exhibits lower efficiency both in time and space complexity. We have noticed this problem and will improve it in future work.
radius
mloU(%)
F-score
3
42.27
0.531
5
43.77
0.551
7
45.41
0.565
9
39.59
0.502
Table 6: Evaluation on the radius r of affinity matrix generation AVSBench-S4 validation set.
Subsequently, conduct ablation studies on the hyper-parameter in the SMR module as shown in Table 6. The parameter r controls the radius of the affinity matrix generation, which determines the scope of interest neighborhood around each point in the coarse sounding maps. In our experiments, we find that setting 7 yields the best results. It's noted that excessive values of may gener- unnecessary relations that can adversely affect the final outcome. Next, we explore the hyper- parameter Eq. 5, specifically focusing on the two background thresholds 3t and f0, which con- trol the Tri-map division. As shown in Table 7, we establish that setting 8) to 0.4 and 83, to 0.55 yields optimal results, respectively. Finally, shown in Table 8, analyze the hyper-parameters of Eq. 7 and Eq. which significantly impact sounding map refinement. The in Eqq 7 for ignoring triv- ial affinity values in the semantic affinity matrix, while the in Eq. controls the iterations the semantic transition matrix. Our parameters search
6
ml)U(%)
F-score
0
43.07
0.545
0.1
44.87
0.564
0.3
45.12
0.561
0.5
45.41
0.565
0.7
44.63
0.554
0.9
13.49
0.223
Table 5: Evaluation the drop probability 8 of Random mask augmentation layer on AVSBench-S4 validation set.
Hlpee-pprrametes. In Table Table 5, we an- MMICL module.
alyze the hyper-parameters The drop probability in the random mask aug- mentation layer controls the probability of visual

---

Ground Truth
VSL
FNAC
USPL(Ours)
Ground Truth
EZ- VSL
FNAC
USPL(Ours)
VGGSS
AVSBench SS4
Figure 4: Qualitative results for USPL training AVSBench-S4 (right) and VGGSS (left). For each dataset, the first column shows annotations overlaid on images, and the following two column show predictions trained on Unsup SSL methods, EZ-VSL (Mo and Morgado, 2022b) and FNAC(Sun et al., 2023). The last column shows our USPL prediction results. Obviously, USPL produces more accurate sounding pixel prediction results than the previous SSL method.
through grid-based evaluation reveals that setting 10 and / 8 yields optimal results for our model.
t
mloU(%)
4
8
12
39.53
34.26
15.19
15.08
3
3
mloU(%)
F-score
40.49
43.68
36.61
13.18
33
0.75
0.6
30.69
0.473
10
40.07
42.51
45.41
24.03
0.65
0.5
37.94
0.527
39.75
41.65
44.95
37.77
0.55
0.4
45.41
0.565
0.45
0.3
41.19
0.492
Table 8: Evaluation on hyper-parameters 8, of the sounding map refinement on AVSBench-S4 validation set, reported in mloU.
Table 7: Evaluation the background thresholds 3n. 3 of Tri-map generation on AVSBench-S4 validation set.
object or predict an approximate region of sound- ing object.
4.4 Qualitative Results
Conclusion
As shown Fig. 4, we show some qualitative com- parisons between previous Unsup SSL. methods, EZ-VSL, FNAC, and our USPL on the AVSBench- S4 test and VGGSS benchmark. As we can observe, compared to Unsup SSL methods, our method gen- crally produces more accurate localization results, As pixel-level method, the predictions of USPL tend to be complete and highly consistent with the shape of the sounding objects, which means, a pre- cise prediction of the object boundaries, while EZ- VSL. and FNAC often locate part of the sounding
In this work, we propose USPI., a novel unsuper- vised approach for pixel SSL. Specifically, we first use the MMICL module to align audio-visual fea- tures and obtain coarse sounding maps. Then, we propose an SMR module to refine the coarse sound- ing maps by the visual semantic affinity. Next, we propose SPS produce final segmentation results. Our approach shows promising performance on pixel SSL tasks, especially achieving new SOTAs on the AVSBench-SS S4 test and VGGSS benchmark.

---

Limitations
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully con- nected crfs. IEFE transactions pattern analysis and machine intelligence, 40(4))834-848
Although USPL behaves well on pixel SSL at sin- gle sound source localization, it is not applicable to localize multiple sound sources in unconstrained videos (Qian et al., 2020), which is still challenge for the community. A potential solution is to de- velop weakly- or semi-supervissd methods. We leave it for future work. Another limitation that we need to address our work is the effective in- tegration temporal information. As video data is inherently a time series, contains a wealth of information that is embedded within the tempo- ral dimension. By solely utilizing single frame, done in previous SSL methods, risk losing valuable temporal context. Therefore, in our future work, will prioritize incorporating temporal in- formation into our approach.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b. A simple framework for contrastive learning of visual representations. In ICML. pages 1597-1607.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In CVPR, pages 770-778.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. Cnn architectures for large- scale audio classification. In ICASSP, pages 131-135. IEEE.
Di Hu, Feiping Nie, and Xuelong Li. 2019. Deep multi- modal clustering for unsupervised audiovisual learn- ing. In CVPR, pages 9248-9257.
References
Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. 2020. Dis- criminative sounding objects localization via self- supervised audiovisual matching. NIPS, 33: 10087.
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. 2020. Self-supervised learn- ing of audio-visual objects from video. In ECCV, pages 224. Springer.
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. 2019. Weakly supervised learning of instance segmentation with inter-pixel relations. In CVPR, pages 2209- 2218.
Alexander Kirillov, Ross Girshick, Kaiming He, and Pi- otr Dollár. 2019, Panoptic feature pyramid networks. In CVPR, pages 3399-6408.
Philipp Krähenbühl and Vladlen Koltun. 201 Efficient inference in fully connected erfs with gaussian edge potentials. NIPS, 24.
Jiwoon Ahn and Suha Kwak. 2018. Learning pixel- level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, pages 4981 49990.
Yan-Bo Lin, Yi-Lin Sung. Jie Lei, Mohit Bansal, and Gedas Bertasius. 2023. Vision transformers are parameter-efficient audio-visual learners. In CVPR, pages 2299-2309.
Relja Arandjelovic and Andrew Zisserman. 2017. Look, listen and learn. In ICCV, pages 609-617.
Relja Arandjelovic and Andrew Zisserman. 2018. Ob- jects that sound. In ECCV, pages 435-451.
Yan-Bo Lin, Hung- Yu Tseng, Ying Lee, Yen- Yu Lin, and Ming-Hsuar Yang. 2021. Exploring cross-video and cross-modality signals for weakly- supervised audio-visual video parsing. NIPS, 111144-11461..
Nikita Araslanov and Stefan Roth. 2020. Single-stage semantic segmentation from image labels. CVPR, pages 4253-4262.
Yusuf Aytar. Carl Vondrick. and Antonio Torralba. 2016. Soundnet: Learning sound representations from unla- beled video. NIPS, 29.
Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. 2022a. Exploiting transformation invariance and equivari- ance for self-supervised sound localisation. ACM MM, pages 3742-3753.
Honglic Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. 2021. Localizing visual sounds the hard way. In CVPR, pages 16867-16876.
Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, and Xiaowei Zhou. 2022b. Visual sound localization in the wild by cross-modal interference erasing. In AAAI, pages 1801-1809.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020a. Vggsound: A large-scale audio- visual dataset. In ICASSP, pages 721-725. IEEE.
Sabarinath Mahadevan, Ali Athar, Aljoša OSep. Se- bastian Hennen, Laura Leal-Taixé, and Bastian Leibe. 2020. Making case for 3d convolutions for object segmentation in videos. arXiv preprint N.Xiv:2008.1151e
Jiang-Chieh Chen, George Papandreou, Iasonas Kokki- nos, Kevin Murphy, and Alan L Yuille. 2017.

---

Yuxin Mao, Jing Zhang, Zhexiong Wan, Yuchao Dai. Aixuan Li, Yunqiu Lv, Xinyu Tian, Deng-Ping Fan, and Nick Barnes. 2021. Transformer transforms salient object detection and camouflaged object de- tection. arXiv preprini (xXi:2204.10122
Shentong Mo and Pedro Morgado. 2022a. closer look at -upeerii-d audio-visual source localiza- tion. In NIPS.
Shentong Mo and Pedro Morgado. 2022b. Localizing visual sounds the easy way. In ECCV. pages 218-234.
Springer.
Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. 2020. Multiple sound sources localization from coarse to fine. In ECCV, pages 292-308.
Lixiang Ru, Yibing Zhan, Baosheng Yu, Bo Du. 2022. Learning affinity from attention: end-to-end wcakly-supervised semantic segmentation with trans- formers. In CVPR, pages 16846-166 16855.
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang. and So Kweon. 2018. Learning to localize sound source in visual scenes. In CVPR, pages 4358- 4366.
Zengjie Song, Yuxi Wang. Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang. 2022. sule-super predictive learning: A negative-free method for sound source localization in visual scenes. In CVPR, pages 3222- 3231.
Weixuan Sun, Jiayi Zhang, Jianyuan Wang, Zheyuan Liu, Yiran Zhong, Tianpeng Feng, Yandong Guo, Yanhao Zhang, and Nick Barnes. 2023. Learning audio-visual source localization via false negative aware contrastive learning. In CVPR, pages 6420- 6429.
Paul Vernaza and Mammohan Chandraker. 2017. Learn- ing random-walk label propagation for weakly- supervised semantic segmentation. In CVPR, pages 7158-7166.
Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li. 2022. Learning in audio-visual context: A re- view, analysis, and new perspective. arXiv preprint arXiv:2208.09579.
Yu and Yang. 2021. Exploring heterogencous clues for weakly-suppervsed audio-visual video pars- ing. In CVPR, pages 1326-1335.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In CVPR, pages 2921-2929.
Jinxing Zhou, Jianyuan Wang. Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Ling- peng Kong, Meng Wang, and Yiran Zhong. 2022. Audio-visual segmentation. In ECCV, pages 386- 403. Springer.


Reviews:
[{"review_id": "eDHbUYZlNU", "reviewer": "Reviewer_3pBn", "summary": "", "questions": "See the questions listed above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our paper. We sincerely appreciate your time and effort in reviewing our work. We would like to address your concern with the following points:\n\n**Q1** : How to determine the values of hyper-parameters, e.g. as described in Section 4.1.\n\n**Answers** : Thanks very much for your valuable feedback. \n\nIn Section 4.1, we list some hyper-parameter settings of USPL . To ensure the optimal performance of the algorithm, we conduct extensive experiments on the AVSBench-S4 dataset, where we explore alternative hyper-parameter combinations. The ablation studies are conducted on AVSBench-S4 train subset and evalated on AVSBench-S4 validation subset. \n\nWe firstly ablate the hyper-parameters in MMICL module. Specifically, we focus on the hyper-parameter temperature  $\\tau$ in Eq. 2, which serves a similar role as the temperature in contrastive learning. This parameter controls the shape of the distribution and plays a crucial role in distinguishing negative samples. Through extensive experiments as shown in Table 1, we find that setting $\\tau$ to 0.07 yields the best result.\n\n**Table 1. Evaluation on the temperature parameter $\\tau$ of MMICL loss on AVSBench-S4 validation set.**\n\n| $\\tau$   | mIoU(%)$\\uparrow$ | F-score$\\uparrow$ |\n| -------- | ----------------- | ----------------- |\n| 0.03     | 44.67             | 0.544             |\n| **0.07** | **45.41**         | **0.565**         |\n| 0.10     | 41.75             | 0.539             |\n| 0.13     | 39.19             | 0.517             |\n\nThen, we investigate the impact of the drop probability $\\delta$ in the random mask augmentation layer as shown in Table 2. This parameter controls the probability of dropping visual features during training, which makes model more robustness. We perform a series of settings and compare the results to the case where no random mask augmentation is used, i.e., $\\delta = 0$. It is observed that using random mask always yields better performance. Moreover, when $\\delta = 0.5$, the model achieves its optimal performance. It is also important to note that excessive drop probability can be detrimental to the model's performance.\n\n**Table 2. Evaluation on the drop probability $\\delta$  of Random mask augmentation layer on AVSBench-S4 validation set.**\n\n| $\\delta$ | mIoU(%)$\\uparrow$ | F-score$\\uparrow$ |\n| -------- | ----------------- | ----------------- |\n| 0        | 43.07             | 0.545             |\n| 0.1      | 44.87             | 0.564             |\n| 0.3      | 45.12             | 0.561             |\n| **0.5**  | **45.41**         | **0.565**         |\n| 0.7      | 44.63             | 0.554             |\n| 0.9      | 13.49             | 0.223             |\n\nSubsequently, we conduct ablation studies on the hyper-parameter in the SMR module as shown in Table 3. The SMR module is responsible for refining the object shape information from coarse sounding maps, which significantly contributes to the improvement of our sounding object localization performance. The parameter $r$ controls the radius of the affinity matrix generation, which determines the scope of interest neighborhood around each point in the coarse sounding maps. In our experiments, we find that setting $r=7$ yields the best results. It's noted that excessive values of $r$ not only consume more computational resources but also generate unnecessary relations that can adversely affect the final outcome.\n\n**Table 3. Evaluation on the radius $r$ of affinity matrix generation on AVSBench-S4 validation set.**\n\n| radius $r$ | mIoU(%)$\\uparrow$ | F-score$\\uparrow$ |\n| ---------- | ----------------- | ----------------- |\n| 3          | 42.27             | 0.531             |\n| 5          | 43.77             | 0.551             |\n| **7**      | **45.41**         | **0.565**         |\n| 9          | 39.59             | 0.502             |\n\nWe explore the hyper-parameter in Eq. 5, specifically focusing on the two background thresholds $\\beta_l$ and $\\beta_h$ which control the Tri-map division. Through extensive experiments in Table 4, we establish that setting $\\beta_l$ to 0.4 and $\\beta_h$ to 0.55 yields optimal results in Eq. 5, respectively.\n\n**Table 4. Evaluation on the background thresholds $\\beta_h$, $\\beta_l$ of Tri-map generation on AVSBench-S4 validation set.**\n\n\n| $\\beta_h$ | $\\beta_l$ | mIoU(%)$\\uparrow$ | F-score$\\uparrow$ |\n| --------- | --------- | ----------------- | ----------------- |\n| 0.75      | 0.6       | 30.69             | 0.473             |\n| 0.65      | 0.5       | 37.94             | 0.527             |\n| **0.55**  | **0.4**   | **45.41**         | **0.565**         |\n| 0.45      | 0.3       | 41.19             | 0.492             |\n\nIn Table 5, we explore the hyper-parameters in Eq. 5, we also delve into the hyper-parameters of Eq. 7 and Eq. 8, which significantly impact sounding map refinement. The $\\beta$ in Eq. 7 is for ignoring trivial affinity values in the semantic affinity matrix, while the $t$ in Eq. 8 controls the iterations of the semantic transition matrix. Our parameters search through grid-based evaluation reveals that setting $\\beta$ = 10 and $t$ = 8 yields optimal results for our model. \n\n**Table 5. Evaluation on the hyper-parameters $\\beta$, $t$ of the sounding map refinement on AVSBench-S4 validation set, reported in mIoU.**\n\n|         |        |       | $t$   |           |       |\n| ------- | ------ | ----- | ----- | --------- | ----- |\n|         |        | 1     | 4     | **8**     | 12    |\n|         | 1      | 39.53 | 34.26 | 15.19     | 15.08 |\n| $\\beta$ | 5      | 40.49 | 43.68 | 36.61     | 13.18 |\n|         | **10** | 40.07 | 42.51 | **45.41** | 24.03 |\n|         | 15     | 39.75 | 41.65 | 44.95     | 37.77 |\n\n\n\n**Q2** : For methods listed in Table 2(in our paper), it helps to include computational and memory costs.\n\n**Answers :** Thanks very much for your valuable suggestion. \n\nHere we discuss the model computation and memory costs as shown in Table 6.\n\nWe calculate the *parameters of model (Params)*,  *floating point operations (FLOPS)*  and *frames per second (FPS)* to evaluate methods' efficiency. The methods listed in Table2(in our paper) are a series of similar methods and we choose the representative work EZ-VSL which also our baseline, showing on the ablation experiment result table. \n\nComparing to the baseline method, our USPL exhibits lower efficiency both in time and space complexity. However, we have noticed the computational efficiency problem and SPS module is proposed for boosting the inference speed. What's more, as part of future work, we aim to improve the efficiency of our method.\n\n**Table 6. Analysis computation and memory cost of USPL evaluation. The results are inference procedure on validation set of AVSBench-S4.**\n\n|                   | mIoU(%)$\\uparrow$ | Params(M)$\\downarrow$ | FLOPS(G)$\\downarrow$ | FPS$\\uparrow$ |\n| :---------------- | ----------------- | --------------------- | :------------------- | ------------- |\n| EZ-VSL (baseline) | 33.85             | 24.14                 | 6.46                 | 15.45         |\n| USPL (SMR)        | 44.97             | 69.64                 | 40.85                | 4.34          |\n| USPL (SPS)        | 45.41             | 68.12                 | 38.61                | 9.94          |\n\n\n\n**Q3** : For the ablation study summarized in Table 3(in our paper), why is the test partition (not the dev partition) of AVSBench-S4 used?\n\n**Answers :** Thanks very much for the question. \n\nIn our paper, Tabel 7 shows the ablation study of various modules in USPL. To keep consistent with previous works, we report the experiments on test subset of AVSBench-S4.  We also evaluate each module on validation subset and we strictly comply with the machine learning training protocol, which means test subset is only used for final test and validation subset is used for hyper-parameter tuning.\n\nWe will report both the validation subset and the test subset results of this ablation study in the final version of our paper.\n\n**Table 7. Ablation study of different modules of the proposed USPL in mIoU. The results are inference procedure on both validation and test subset of AVSBench-S4.**\n\n| Methods          | MMICL | SMR  | SPS  | *val*     | *test*    |\n| ---------------- | ----- | ---- | ---- | --------- | --------- |\n| EZ-VSL(Baseline) |       |      |      | 33.85     | 34.01     |\n|                  | √     |      |      | 34.09     | 35.52     |\n| USPL(Ours)       | √     | √    |      | 43.72     | 44.01     |\n|                  | √     | √    | √    | **45.41** | **47.19** |\n\n\n\n**Q4 :** Line 521 mentions \"Fig. 3\", and I think authors meant \"Fig. 4\" there?\n\n**Answers :** Thanks very much for pointing out our issue. \n\nWe sincerely apologize for mistakenly writing \"Fig. 4\" to \"Fig. 3\". We have thoroughly check and correct it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's with broad interests to investigate pixel sound source location in an unsupervised manner, which mitigates the challenges of obtaining large volume of annotations. Overall, this paper clearly presents the proposed methodology, with well-designed experimental design and ablation studies. ", "reasons_to_reject": "I think more details could be added to strengthen the experimental session, including:\n1. How to determine the values of hyper-parameters, e.g. as described in Section 4.1.\n2. For methods listed in Table 2, it helps to include computational and memory costs.\n3. For the ablation study summarized in Table 3, why is the test partition (not the dev partition) of AVSBench-S4 used?\n\n", "questions_to_authors": "See the questions listed above.", "paper_topic_and_main_contributions": "In this paper authors propose Unsupervised Sounding Pixel Learning (USPL) to improve cross-model pixel-level sounding source localization. USPL consists of modules for mask augmentation based multi-instance contrastive learning (MMICL), Sounding Map Refinement (SMR), Sounding Pixel Segmentation (SPS) etc. Experimental results show the effectiveness of the proposed approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 521 mentions \"Fig. 3\", and I think authors meant \"Fig. 4\" there?"}, "extra_text": {}}, {"review_id": "Bju3SFJPa2", "reviewer": "Reviewer_A2BU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for your insightful feedback on our paper. We sincerely appreciate your time and effort in reviewing our work. In response to your concerns, we would like to provide the following clarifications:\n\n**Q1** : The efficiency of the SMR module appears to be low. Although the authors suggest using SPS to bypass the SMR during the evaluation stage, there is no corresponding discussion in the experimental section. I hope to see a comparison of the time and space complexity with and without SPS during evaluation.\n\n**Answers :** Thanks very much for your valuable suggestion. \n\nHere we discuss the model computation and memory costs as shown in Table 1.\n\nIn our proposed method, the SMR and SPS modules play a crucial role. SMR module refines the sounding maps by leveraging visual semantic affinity, resulting in fine-grained sounding maps. However, it is noted that SMR exhibits low efficiency. Besides, SPS module serves as a lightweight evaluation module. We calculate the *parameters of model (Params)*,  *floating point operations (FLOPS)*  and *frames per second (FPS)* to evaluate methods' efficiency. We find that using SPS to bypass SMR may slightly reduce space complexity and significantly decreases time complexity, which leading to better performance.\n\n**Table 1. Analysis computation and memory cost of USPL evaluation. The results are inference procedure on validation set of AVSBench-S4.**\n\n|            | mIoU(%)$\\uparrow$ | Params(M)$\\downarrow$ | FLOPS(G)$\\downarrow$ | FPS$\\uparrow$ |\n| :--------- | ----------------- | --------------------- | :------------------- | ------------- |\n| USPL (SMR) | 44.97             | 69.64                 | 40.85                | 4.34          |\n| USPL (SPS) | 45.41             | 68.12                 | 38.61                | 9.94          |\n\n\n\n**Q2** : Since the SMR module significantly contributes to the performance improvement, has there been any exploration of the hyperparameters used in Eq. 5, 7, and 8?\n\n**Answers :** Thanks very much for the question. \n\nSMR(Sounding Map Refinement) module contributes significantly to our sounding object localization performance improvement. Here we will explore the hyper-parameters involved in SMR as shown in Table 2. We first explore the hyper-parameters in Eq. 5, specifically focusing on the two background thresholds $\\beta_l$ and $\\beta_h$ which control the Tri-map division.  We set a list of hyper-parameters finding that $\\beta_l$ = 0.4 and $\\beta_h$ = 0.55 are best in Eq. 5, respectively. \n\n**Table 2. Evaluation on the background thresholds $\\beta_h$, $\\beta_l$ of Tri-map generation on AVSBench-S4 validation.**\n\n| $\\beta_h$ | $\\beta_l$ | mIoU(%)$\\uparrow$ | F-score$\\uparrow$ |\n| --------- | --------- | ----------------- | ----------------- |\n| 0.7       | 0.6       | 30.69             | 0.473             |\n| 0.6       | 0.5       | 37.94             | 0.527             |\n| **0.5**   | **0.4**   | **45.41**         | **0.565**         |\n| 0.4       | 0.3       | 41.19             | 0.492             |\n\nAfter exploring the hyper-parameters in Eq. 5, we also delve into the hyper-parameters of Eq. 7 and Eq. 8, which significantly impact sounding map refinement. The $\\beta$ parameter in Eq. 7 serves as a hyper-parameter for ignoring trivial affinity values in the semantic affinity matrix, while the $t$ parameter in Eq. 8 controls the iterations of the semantic transition matrix. Our parameters search through grid-based evaluation reveals that setting $\\beta$ = 10 in Eq. 7 and $t$ = 8 in Eq. 8 yields optimal results for our model as shown in Table 3. \n\n**Table 3. Evaluation on the hyper-parameters $\\beta$, $t$ of the sounding map refinement on AVSBench-S4 validation, reported in mIoU.**\n\n|         |        |       | $t$   |           |       |\n| ------- | ------ | ----- | ----- | --------- | ----- |\n|         |        | 1     | 4     | **8**     | 12    |\n|         | 1      | 39.53 | 34.26 | 15.19     | 15.08 |\n| $\\beta$ | 5      | 40.49 | 43.68 | 36.61     | 13.18 |\n|         | **10** | 40.07 | 42.51 | **45.41** | 24.03 |\n|         | 15     | 39.75 | 41.65 | 44.95     | 37.77 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed Sounding Map Refinement (SMR) module is novel and reasonable to me.\n\n2. The experimental results indicate that the proposed method outperforms previous unsupervised methods.\n\n3. The paper is well-written and easy to follow.", "reasons_to_reject": "1. The efficiency of the SMR module appears to be low. Although the authors suggest using SPS to bypass the SMR during the evaluation stage, there is no corresponding discussion in the experimental section. I hope to see a comparison of the time and space complexity with and without SPS during evaluation.\n\n2. Since the SMR module significantly contributes to the performance improvement, has there been any exploration of the hyperparameters used in Eq. 5, 7, and 8?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to generate precise pixel-level object maps from videos under the unsupervised setting. Specifically, the authors propose a two-step approach: initially aligning audio-visual features coarsely using the MMICL module, followed by refining the coarse maps using an unsupervised SMR module. Additionally, they incorporate a sounding pixel segmentation (SPS) module for lightweight evaluation and improved performance. The experimental results demonstrate the effectiveness of the proposed methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fQIbZAtoAX", "reviewer": "Reviewer_c1YV", "summary": "", "questions": "To me, the motivation of introduction of the SPS module remains unclear. What do you mean by ‘Although the SMR produces fine sounding maps, it is too heavy for evaluation’? More explanation may be needed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our paper. We sincerely appreciate your time and effort in reviewing our work. We would like to address your concern with the following points:\n\n**Q1** : To me, the motivation of introduction of the SPS module remains unclear. What do you mean by ‘Although the SMR produces fine sounding maps, it is too heavy for evaluation’? More explanation may be needed.\n\n**Answers :** Thanks very much for your valuable suggestion. \n\nHere we discuss the model computation and memory costs to explain the motivation of SPS as shown in Table 1.\n\nIn our proposed method, the SMR and SPS modules play a crucial role. SMR module refines the sounding maps by leveraging visual semantic affinity, resulting in fine-grained sounding maps. However, it is noted that SMR exhibits low efficiency. Therefore, SPS module serves as a lightweight evaluation module. We calculate the *parameters of model (Params)*,  *floating point operations (FLOPS)*  and *frames per second (FPS)* to evaluate methods' efficiency. We find that using SPS to bypass SMR could slightly reduce space complexity and significantly decreases time complexity, which leading to better performance.\n\n**Table 1. Analysis computation and memory cost of USPL evaluation. The results are inference procedure on validation set of AVSBench-S4.**\n\n|            | mIoU(%)$\\uparrow$ | Params(M)$\\downarrow$ | Flops(G)$\\downarrow$ | FPS$\\uparrow$ |\n| ---------- | ----------------- | --------------------- | -------------------- | ------------- |\n| USPL (SMR) | 44.97             | 69.64                 | 40.85                | 4.34          |\n| USPL (SPS) | 45.41             | 68.12                 | 38.61                | 9.94          |\n\n\n\n**Q2**  : In the caption of Fig. 2, SOS should be SPS.\n\n**Answers:** Thanks very much for pointing out our issue. \n\nWe sincerely apologize for mistakenly writing \"SPS\" to \"SOS\" in the caption of Fig. 2. We have thoroughly check and correct it.\n\n\n\n**Q3** : In eq 11 and other places, why mmicl and seg are italic while aff is not?\n\n**Answers:** Thank you very much for pointing out our problem. \n\nWe sincerely apologize for mistakenly font using. We have thoroughly check and correct it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. To my knowledge, the task of Unsupervised Sounding Pixel Learning (USPL) is first explored by this paper. This is a novel, interesting and challenging task which may attract more researchers.\n2. Experiments on public datasets show that the proposed method considerably outperforms current systems and achieves SOTA, demonstrating the effectiveness of the paper.", "reasons_to_reject": "1. I wonder whether the topic of this paper is suitable for EMNLP, since the paper focuses on sounding pixel location, which has nothing to do with NLP.\n2. To me, the motivation of introduction of the SPS module remains unclear. What do you mean by ‘Although the SMR produces fine sounding maps, it is too heavy for evaluation’? More explanation may be needed.", "questions_to_authors": "To me, the motivation of introduction of the SPS module remains unclear. What do you mean by ‘Although the SMR produces fine sounding maps, it is too heavy for evaluation’? More explanation may be needed.", "paper_topic_and_main_contributions": "This paper focuses on a novel and interesting task, namely, Unsupervised Sounding Pixel Learning (USPL). The authors proposed a method with a mask augmentation based multi-instance contrastive learning module, an Unsupervised Sounding Map Refinement (SMR) module, and a Sounding Pixel Segmentation (SPS) module. Experiments on public datasets show that the proposed method considerably outperforms current systems and achieves SOTA.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In the caption of Fig. 2, SOS should be SPS.\n2. In eq 11 and other places, why mmicl and seg are italic while aff is not?"}, "extra_text": {}}, {"review_id": "z0wtKwQzRa", "reviewer": "Reviewer_fozg", "summary": "", "questions": "A. Have you attempted to leverage the temporal information between frames in your analysis? Could this approach provide any benefits?\n\nB. Have you assessed the contribution of audio to the model? How does the performance vary when audio information is not provided? Can the model still achieve satisfactory performance by merely predicting the mask of the salient object in the image?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review of our paper. Your attention to detail and expertise have contributed significantly to the improvement of our work. We are thankful for your support and look forward to incorporating your suggestions into our future work.\n\n**Q1** : While the paper presents a detailed description of the proposed method, it lacks a deeper theoretical analysis and explanation of the underlying principles supporting the effectiveness of the approach.\n\n**Answers :** Thank you very much for valuable suggestion. \n\nWe will add deeper theoretical analysis and explanation supporting to our method.\n\nOur method is based on two facts. One is that huge numbers of videos exist, and it is hard to apply traditional supervised segmentation methods on these huge video data because of heavy manual efforts. Therefore, we propose a totaly unsupervised method USPL to segment the object in videos. Another fact is that visual events are usually accompanied by sounds in our daily lives. There are a few tries which construct a model to learn the correspondence between visual scene and the sound as the self-supervised training strategy. These methods also find that after self-supervised training, model has the ability to locate the sounding related objects by cross modal attention. \n\nHowever, the ability of sounding localization learnt by self-supervised training seems not strong enough. Our USPL is based on the correspondence self-supervised training and considers the learnt sounding map as a initial sounding area. Then USPL applys a series of refine steps to get pixel-level sounding map results. \n\nThe most important step of our refine steps is SMR module which employs the visual semantic affinity learning to explore inter-pixel relations of adjacent coordinate features. It contributes to recovering the boundary of coarse sounding maps, obtaining fine sounding maps. In short, the SMR is a kind of affinity methods which have been used to enhance the quality of semantic segmentation. For example, in [1], CNNs for semantic segmentation are incorporated with a differentiable module computing a semantic affinity matrix of pixels, and trained in an end-to-end manner with full supervision.  In our SMR, we predicts a high-dimensional embedding vector for each pixel, and the affinity between a pair of pixels is defined as the similarity between their embedding vectors.  We use the learnt affinity to propagate coarse sounding map to discover the entire instance areas with accurate boundaries.\n\n  \n\n[1] G. Bertasius, L. Torresani, S. X. Yu, and J. Shi. Convolutional random walk networks for semantic image segmentation. In CVPR, 2017  \n\n\n\n**Q2** : Have you attempted to leverage the temporal information between frames in your analysis? Could this approach provide any benefits?\n\n**Answers :** Thanks very much for the question. \n\nTo leverage the temporal information between frames, we here realize it by replacing the visual backbone from ResNet to C3D which extracts temporal feature. On the AVSBench-S4 validation dataset, our model achieves 0.37% mIoU and 0.06 F-score improvement after employing temporal input as shown in Table 1. This means temporal information is very explorative for our method and we believe temporal information will be a great help to our method after more tries.\n\n**Table 1. Analysis importing temporal information to the proposed USPL. The results are inference procedure on validation set of AVSBench-S4.**\n\n| Methods                   | mIoU(%)$\\uparrow$ | F-score$\\uparrow$ |\n| ------------------------- | ----------------- | ----------------- |\n| USPL w/ sigle-frame input | 45.41             | 0.565             |\n| USPL w/ sequence input    | 45.78             | 0.571             |\n\n\n\n**Q3** : Have you assessed the contribution of audio to the model? How does the performance vary when audio information is not provided? Can the model still achieve satisfactory performance by merely predicting the mask of the salient object in the image?\n\n**Answers :** Thanks very much for the question. \n\nDuring the training phase, audio modality serves as a core requirements for our method. As an unsupervised approach, USPL learns the correspondence between visual scene and the sound as a self-supervised training strategy. The correspondence learning of audio and visual process is so much essential. If without it, our USPL can't produce the initial sounding maps and also can't do the next refine processes. Therefore, our USPL must have both audio and visual modalities to work in training and these correspondence two modalities are naturally exist in video data.\n\nDuring the evaluation phase, audio is not the necessary modality for our method. Because in our evaluation stage, we only use SPS module, and audio modality only participants in the cross-modal fusion to improve the performance. In our ablation experiment, without audio fusion in SPS, the mIoU on AVSBench-S4 validation and test subset only drops 0.05% and  1.46% respectively. This proves that our model still achieves satisfactory performance by merely predicting the mask of the salient object in the image.\n\nTable 2 which analyses the audio modality contribution to the proposed USPL is listed here: \n\n**Table 2. Analysis the audio modality contribution to the proposed USPL. The results are inference procedure on both validation and test subset of AVSBench-S4.**\n\n| mIoU(%)$\\uparrow$     | *val* | *test* |\n| --------------------- | ----- | ------ |\n| evaluation w/o audio  | 45.37 | 45.73  |\n| evaluation w/   audio | 45.41 | 47.19  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n1. Novel approach: The paper proposes an unsupervised method USPL for pixel-level sound source localization. It addresses the challenges of learning accurate pixel-level sounding maps and fine-grained sound source localization in an unsupervised paradigm.\n\n2. Multi-stage approach: The paper includes three main modules - mask-augmentation based multi-instance contrastive learning, unsupervised sounding map refinement, and sounding pixel segmentation. This multi-stage approach allows for the alignment of audio-visual features, refinement of coarse sounding maps, and generation of fine sounding maps.\n\n3. Experimental results: The proposed USPL method outperforms previous state-of-the-art unsupervised methods in terms of mIoU and F-score. It achieves a significant improvement in sound source localization accuracy on both the AVSBench-S4 and VGGSound datasets.\n\nBenefits to the community:\n\n1. Unsupervised sound source localization: The USPL method presents a valuable contribution to the community by addressing the challenge of unsupervised sound source localization. By developing an approach that does not rely on manual annotations, it provides a more efficient and scalable solution for this task.\n\n2. Accurate pixel-level localization: The USPL method achieves accurate pixel-level sound source localization, which is crucial for various applications, such as audio event detection, speech recognition, and audio-visual scene understanding. It provides researchers and practitioners with a reliable tool for extracting precise information from audio-visual data.\n\n3. Superior performance: The experimental results demonstrate the superiority of the proposed USPL method compared to existing unsupervised methods. The improved localization accuracy can benefit researchers in developing more accurate and robust models for various audio-related tasks.", "reasons_to_reject": "While the paper presents a detailed description of the proposed method, it lacks a deeper theoretical analysis and explanation of the underlying principles supporting the effectiveness of the approach.", "questions_to_authors": "A. Have you attempted to leverage the temporal information between frames in your analysis? Could this approach provide any benefits?\n\nB. Have you assessed the contribution of audio to the model? How does the performance vary when audio information is not provided? Can the model still achieve satisfactory performance by merely predicting the mask of the salient object in the image?", "paper_topic_and_main_contributions": "The paper addresses the problem of unsupervised sound source localization in videos. The authors propose an approach called Unsupervised Sounding Pixel Learning (USPL) that aims to accurately localize sounding objects in a given audio-visual scene without any annotations. They propose three main modules: (1) Mask augmentation based multi-instance contrastive learning to obtain coarse sounding maps, (2) Unsupervised Sounding Map Refinement (SMR) module to refine the coarse maps and recover object boundary information, and (3) Sounding Pixel Segmentation (SPS) module to produce the final pixel-level segmentation results.\n\nThe main contributions of the paper are as follows:\n\n1. The authors propose a novel approach, USPL, for unsupervised pixel-level sound source localization in videos. They address the limitation of previous unsupervised methods that focus on coarse localization by aiming to produce fine-grained pixel-level localization results.\n\n2. The proposed method includes a mask augmentation based multi-instance contrastive learning module, which alleviates the problem of overfitting and sub-optimization in previous SSL methods. This module aligns audio-visual features to obtain coarse sounding maps.\n\n3. The authors introduce the Unsupervised SMR module that leverages visual semantic affinity to recover the sounding object boundary and produces fine sounding maps. This module contributes to the recovery of object boundary information from the coarse maps obtained in the previous step.\n\n4. The SPS module is presented to realize audio-supervised semantic segmentation. This module improves the training performance of SMR to predict more accurate pixel-level sounding localization results.\n\nThe authors conduct extensive experiments on the AVSBench-S4 and VGGSound datasets and show improved performance compared to previous state-of-the-art unsupervised methods. The results demonstrate the superiority of the proposed approach in terms of mIoU and cIoU metrics.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies unsupervised sound source localization from multimodal input (images of objects that make sound, along with the corresponding sound waveforms they produce). 

Pros (from reviewers):
The paper studies an interesting and well-motivated problem
The presentation of the experiments in the paper is clear
The experimental results demonstrate that the proposed method outperforms previous models
Some of the model components are technically novel (Sounding Map Refinement)

Cons (from reviewers):
The work may or may not be a good fit for EMNLP (it does deal with multimodality and sound, but not with language)
The motivation and reasoning behind some of the model components (and why they work) is not adequately explained
The experiments do not explore computational or memory costs
The paper did not fully explore the hyperparameter tuning of its proposed components
Decision: Accept-Main