Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization
Authors: Yuxin Guo
Shijie Ma
Hu Su
Zhiqing Wang
Yuhao Zhao
Wei Zou
Siyang Sun
Yun Zheng
Venue: NeurIPS 2023 poster (2023)
Keywords: Audio-Visual Learning
Audio-Visual Source Localization
Semi-Supervised Learning
Multimodal Learning
TL;DR: We propose a novel semi-supervised learning framework for audio-visual source localization to achieve remarkable improvements with limited positional annotations.

Abstract: Audio-Visual Source Localization (AVSL) aims to locate sounding objects within video frames given the paired audio clips. Existing methods predominantly rely on self-supervised contrastive learning of audio-visual correspondence. Without any bounding-box annotations, they struggle to achieve precise localization, especially for small objects, and suffer from blurry boundaries and false positives. Moreover, the naive semi-supervised method is poor in effectively utilizing the abundance of unlabeled audio-visual pairs. In this paper, we propose a novel Semi-Supervised Learning framework for AVSL, namely Dual Mean-Teacher (DMT), comprising two teacher-student structures to circumvent the confirmation bias issue. Specifically, two teachers, pre-trained on limited labeled data, are employed to filter out noisy samples via the consensus between their predictions, and then generate high-quality pseudo-labels by intersecting their confidence maps. The optimal utilization of both labeled and unlabeled data combined with this unbiased framework enable DMT to outperform current state-of-the-art methods by a large margin, with CIoU of $\textbf{90.4\%}$ and $\textbf{48.8\%}$ on Flickr-SoundNet and VGG-Sound Source, obtaining $\textbf{8.9\%}$ and $\textbf{9.6\%}$ improvements respectively, given only $3\%$ of data positional-annotated. We also extend our framework to some existing AVSL methods and consistently boost their performance. Our code is publicly available at https://github.com/gyx-gloria/DMT.

Dual Mean-Teachers An Unbiased Semi-Supervised Framework for Audio-Visum Source Localization
Yuxin 12.3, Shijie Ma 1.2 Hu Sul.2, Zhiqing Wang Yuhao Zhao! Wei Zou 1.2+ Siyang Sun Yun Zheng"
School Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China State Key Laboratory Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation of Chinese Academy of Sciences, Beijing, China
DAMO Academy, Alibaba Group (guoyuxin2021, wei wei.zou] zou) ac
Abstract
Audio- Visual Source Localization (AVSL. aims to locate sounding objects within video frames given the paired audio clips. Existing methods predominantly rely self-supervised contrastive learning of audio-visual correspondence, Without any bounding-box annotations, they struggle achieve precise localization, especially for small objects, and suffer from blurry boundaries and false positives. Moreover, the naive supervise method is poor in fully leveraging the information of abundant unlabeled data. In this paper, we propose a novel cumi-supereised learn- ing framework for AVSL, namely Dual Mean-Teacher (DMT), comprising two teacher-student structures to circumvent the confirmation bias issue. Specifically, teachers, pre-trained samples via the consensus between their predictions, and then generate high-quality pseudo-labels by intersecting their confidence maps. The sufficient utilization of both labeled and unlabeled data and the proposed unbiased framework enable DMT outperform current state-of-the-arr methods by large margin, with CloU of 90.4% % and 48.8% on Flickr-SonndNet and VGG-Sound Source, obtaining 8.9%, 9.6% and 4.6% 6.4% improvements over self- and semi-supervised methods re- spectively, given only < 3% some existing AVSL methods and consistently boost their performance. Our code is available at https:t///// // github..... com/ gyx- gloria/DD
limited labeled data, employed to filter out noisy
We also extend our framework
1 Introduction
Visual and auditory perception is crucial for observing the world. When we hear a sound, our brain will extract semantic information and locate the sounding source. In this work, we focus Audio-Visun Source Localization (AVSL) 21, with the purpose of accurately locating sounding objects in frames based on their paired audio clips. Beyond this scope, AVSL also plays crucial role in many downstream tasks including environmental perception [3]. navigation [4] [5,, sound separation 6 7 and event localization
g Therefore, accurate localization is of utmost importance.
the conventional paradigm is to employ self-supervised contrastive learning based on audio visual correspondence. However, most of them suffer from some serious challenges. From the performance perspective, there are issues such blurry boundaries, inability to converge to specific objects, and the predicted sounding regions that are too large to accurately locate objects, especially small objects. terms the learning stage, a single model alone is unable recognize and filter out false positives, i.e., noisy samples with no visible sounding sources, which could affect the entire learning process the model.
In the literature of AVSL 9
10
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Ground Trush
EZVSL
SLAVC
DMT
Ground Truth EZVSL
SLAVC
DMT
Performance of Existing Methods
500
small object up
90
75
blurry boundaries mi
82
Clal
too much foreground
R12
114
inablity of specific site
Flickr 144k
VGG- 144K
Figure 1: Comparison of existing Audio Visual Source Localization (AVSL) methods and the proposed Dual Mean-Teacher (DMT). Left: DMT has greatly addressed severe issues including inaccurate small object localization, blurry boundaries, and instability. Right: DMT outperforms previous by a large margin on Flickr and VGG-ss datasets.
In essence, AVSL is a dense prediction task, which not be directly accomplished from shared global image representation [12). requiring models to capture fine local features in order to accurately predict object locations, i.e., achieving precise pixel-level localization is not feasible without positional annotations. Unluckily, the number of samples with location labels is extremely limited. As result, we resort to Semi- Suni--upee Learning
[3 (SSL) fully leverage the labeled data.
Considering that self-supervised AVSL is not fully learnable, Attention 10k [4 [3] extended the self-super model to SSL model by directly appending supervised loss labeled data, which is the first semi-supervised attempt in the field. Nevertheless, simply leveraging labeled data might lead to overfitting and neglect to fully harness the underlying unlabeled data. Given these issues, we resort to pseudo-labeling [16]. However, directly introducing pseudo-labeling could lead to confirmation bias 11 which cannot be adequately rectified by a single model.
To tackle these challenges, we break away from traditional self-supervised learning and propose more sophisticated Sepi-Superrvied Audio-Visual Source Localization (SS-AVSL) framework. called Dual Mean-Teacher (DMT), which adopts a double teacher-student structure in a two-stage training manner. We consider previous AVSL methods as single student unit. To fully leverage positional annotations and training data, we extend to a classic semi-supe framework Mean- Teacher 118. To address the issue of confirmation bias, we expand it into dual independent teacher-student structure with designed modules of Noise Filtering, Intersections of Exeudo-Labell (IPL., as shown in Figure 2 Specifically, teachers pre-trained in Warm-Uj Up stage, establishing a solid foundation, in the subsequent Sobiased-Learninng Stage, dual teachers filter out noisy samples and rectify pseudo-labels. In more detail, the Noise Filtering module effectively rejects noise samples by leveraging consensus, i.e., agreement, between dual teachers, ensuring high-quality training data, then IPL module generates precise pseudo-labels by intersecting the predictions from both teachers. DMT eliminates the influence of confirmation bias by rejecting noisy samples and improving the quality of pseudo- labels, which effectively tackles the issues of false positives and greatly improves localization performance.
limited amount of labeled data
In summary, our method contributes the following three aspects. Firstly, we introduce novel unbiased framework based on a pseudo-labeling mechanism for semi-supervised AVSI, which could maximize the utilization of both labeled and unlabeled data, effectively address the challenge of limited annotated data, and mitigate the issue of confirmation bias. Moreover, compared to existing approaches, DMT achieves much remarkable localization performance, with better small object localization and stronger generalization capability, which significantly elevate the performance of current methods. Finally, DMT can be summarized as semi- sumevviueeeieed learning paradigm and could be combined with existing tweakly-superving AVSL methods to consistently boost their performance.
2 Related Works
Semi-Supervised Learning. Semi-Supervised Learning (SSL) 13 of labeled data to unlock the potential of unlabeled data for better model learning. One line of work relies on consistency regularization [18 20 21 to encourage models to behave similarly under
19
leverages small amount
different perturbations. An orthogonal idea is generate high-quality pscudo-labels [16] 22 on

---

unlabeled data to retrain models for better performance. The quality of pseudo-labed labels is crucial.
Current methods [3 24 25 combine the above two paradigms achieve remarkable results.
Audio-Visura Source Localization. The key to Audio Visual Sound Localization (AVSL) en- deavors to establish the correspondence between visual objects and their corresponding sounds by contrastive learning [26] [7]. Most existing methods predominantly utilize self self-supe or weakly-supervised approaches (for all of them employ pre-trained backbone). Some classical works, such as Attention I Ok [14 DMC (2), LVS 28 EZVSL [9] SSPI. SSL-TTH [30] achieve improving performance over time. Other methods like DSOL (31, CoarsetoFine [2), mix and
localize [3 and AVGN [ pay attention to multi-source localization. In addition, some studies also address the issue of false positives and false negatives in supervseee learning. For example, false positives and effectively overcomes the overfitting issue. IER (34 proposes a label-free method that targets the suppression of non-sounding objects, while Robust 35 considers both false positive and false negative issues. AVID [36] detects false negatives by defining sets of positive and negative samples via cross modal agreement. Source separation [37 38 and
SLAVC [0] focuses
generative models [39 also achieve good results. However, most AVSL methods exhibit subpar performance in the absence of annotation.
Semi-Supervised Learning in Localization. Semi-Superviscde Object Detection (SSOD) is one of the few applications of SSL in the localization field. The majority of SSOD methods, such as [40
41 42 utilize pseudo-labeling to enhance the localization performance. Moreover, some works 43 focus on the confirmation bias in SSOD. Similar to object detection, AVSL is a pixel-
like
wise dense prediction task, heavily reliant high-quality pseudo-labels. Attention10k [14 15 is the first SS-AVSL work. It extends a self-supervised model to a semi-supervised framework by simply adding supervised loss, aiming at fixing the false conclusions generated by weakly -upervssd methods. However, this naive method may lead to overfitting and neglects the full utilization of unlabeled data. In contrast, we introduce novel SS-AVSL framework based on pseudo-label mechanism, which can address confirmation bias and maximize the utilization of both labeled and unlabeled data, to achieve stronger localization performance.
3 Background
Problem Definition. Audio- Visual Source Localization (AVSL) aims to accurately locate the sound source within a given visual scene. We denote audio-visual pairs as (as, vz), where a, and es represent the audio and visual modality, respectively. The objective generate a pixel-wise confidence map P indicating the location of the sound source.
Contrastive Learning in AVSL. Self-supervised AVSL methods commonly leverage audio-visure correspondence to maximize the similarity between frames and their corresponding audio clips (positive pairs) while minimizing the similarity among unpaired ones (negative pairs):
Lunsap -B(a,,e)>D. log 2j=1 (x (g(()), ((;)) /T.) + log exp(s(f(v)), ((:)))/)
(xp(((((a).f())))
E5=1 exp (8 (J(vv) g(a;)) /T))
where D, are unlabeled datasets, g(.) and f()) are audio and visual feature extractors. ti is the temperature coefficient. s(-) is consistency matching criterion. The predicted map P; is typically calculated with cosine similarity sim(-) to represent the confidence of the presence of sounding objects:
((aa),(())) P. sim(g(a)), [[o(a.)] -1f(()
Learning with (Pseudo) Labels in AVSL When labeled data are available, one could apply supervised loss directly to learn to localize:
Cup E.D P:).
where G. could be the ground truth or generated pseudo-labels, both 9; and P; are in the form of binary confidence map. H (-,-) -) is cross-entropy function across the two-dimensional spatial axes.
3

---

Corssnnuu inster
NOIS REJECT
Intersection
Weak Augment
Tost
Filter
Pleuud
Teatreer B
IPL
ACCEPT
IPL
Teachers
pos ponudo-laaels
Noise Filtering
IPL
Supervision
Student A
Strong Augment
Network
cusine simlaitt
Fittred
Labeled
Student
Tramm
Unsupervtsed
Unabeltd
Mixed
Students
Dual Mean-Teacher
audio
Network
AVSL Pipetine
Learming
Figure 2: Overview of the proposed Dual Mean-Teacher framework. Left: Overall learning process of dual teacher-student pairs, two students are guided by both ground-truth labeled data and filtered unlabeled data with the Intersection of Pseudo-Labels (IPL). Upper-right: Details of Noise Filtering and IPL. Dual teachers reject noise samples based on their consensus and generate pseudo-labele on filtered data. Lower-right: Details of AVSL pipeline. Students are learned through contrastive learning and predict confidence maps for supervised learning with (pseudo) labels.
4 Dual Mean-Teacher
Overview. In this section, we mainly describe Dual learning process. Specifically, in the Warm-Up Stage (Section 4.1), two teachers are pre-trained on bounding-bon annotated data obtain a stable initialization. In the subsequent Jubiased-Learning Stage (Section 4.2 the Noise Filtering module and Intersection of Pseudo-Lafe (IPL.) module collectively filter out noisy samples and generate high-quality pseudo-labels by two teachers to guide students' training, teachers are in turn updated by exponential moving average (EMA) of students.
(DMT) in the order of the
From a unified perspective, existing AVSL methods be viewed single student, which is later expanded into the semi-supervised classical framework Mean-Teacher [18), in order to fully utilize limited labeled data. To effectively address the confirmation bias issue, further extend it double teacher-student framework, as shown in Figure 2 DMT adopts teacher-student architecture, where each teacher and student contains two independent AVSL pipelines for audio-visual contrastive learming and generating localization results. Teachers provide students with stable unlabeled samples after Noise Filtering and their generated IPL. Students pass back the parameters to the teachers.
General Notations. Cross-entrops function -:: and two feature extractors f(-) and g(-) are already discussed in Section B By default, subscript indicates the i-th sample, while superscript A. B denote two teacher-student pairs with and s indicating teacher and student, respectively. D, and Dr labeled and unlabeled datasets. Gi and P, ground-truth and predicted confidence maps, both with size of H x W. We apply strong A(-) or weak ax-) augmentation on visual inputs.
4.1 Warm-Up Stage
The quality of pseudo-labels is crucial to SSL, especially for localization tasks. Before generating pseudo-labels, we first pre-train dual teachers with bounding-box annotated data achieve prelim- inary localization performance. In order to avoid overfitting, we apply strong augmentation and get augmented labeled dataset D;- = {(a A(vx)) (. Gi). After extracting visual features f ('(A(;;) and auditory features ge (a.), the predicted map Pi can be obtained by Eq. (2) Then we utilize bounding-boo box annotations G, as supervision:
Lwarm-Up E(a. P!).
4.2
Unbiased-Learne Stage
Noise Filtering. To mitigate confirmation bias, it is crucial filter out noisy samples that are more likely to be false positives. As depicted in Figure 2 two predicted maps of the same sample are generated by dual teachers. It is clear that samples with higher reliability can be identified when the two predicted maps exhibit higher similarity, i.e., there is more agreement and consensus between dual teachers, then the sample is reserved for pseudo-labelling, Conversely, when there is a significant discrepancy between the two maps, the sample will be considered as false positive, such as frames without distinguishable sound objects or sounds that cannot be accurately represented by a bounding box (e.g., wind sounds), such samples are rejected and discarded.

---

Intersection of Pseudo-Labels (IPL), By intersecting the foreground regions of two predicted maps on the filtered samples, one can generate positional pseudo-labels, named Intersection of Pseudo-Labels (IPL., to guide students' learning. With the pre-defined foreground threshold 6, two predicted maps P..AA p. could be transferred to binary maps M and M. Weak augmentation a(-) is employed for teachers to generate high-quality pseudo-labels:
P = sim(g' f(()))()(v)))))
(5)
M = 1P: 2 8).
(6)
We adopt the Intersection over Union (IoU) metric to quantify the similarity between the two maps M.A.A. M If the IoU score exceeds the threshold т, the sample will be accepted, and the intersection of those two maps will be generated as its pseudo-label (IPL). Otherwise, will be filtered out as noise sample.
D' = {(a,; ToU(MA .M:B)2 2 t, V(ar, 14) E D.).
IPL( us) = M M
(8)
The newly selected unlabeled dataset is applied to the student model along with the corresponding high-quality IPL.
Students Learning without bias. and new unlabeled datasets. Both ground-truth annotations and high-quality IPL are employed to train the student models:
To suppress confirmation bias more effectively, we mix labeled
(ai, 0z) E Dr Dmic= = Dj DIUD' = {(a v.) where 9. { IPL v.) if (a, B.) E D
Gi
(9)
In addition, we incorporate consistency regularization 45 in the semi-supervised learning process. Specifically, for a given sample, we obtain IPL from the teachers on weakly augmented images while strong augmentations are applied for samples of students. By enforcing consistency between IPL and students' predictions, DMT could be more stable with better generalization ability.
P si (*((;))))
(10)
Suup EENDm.. H
(11)
Similar to the AVSL method mentioned in Section 3 students are also trained by audio-visual correspondence of contrastive learning loss. Here, we introduce an attention module to add attention to the sounding region in the frame:
exp (Pi(s, v))
(uss(1))
f(w)).
(12)
Exsy exp (P.(x.)) v))
Then, the full semi supervvid loss could be derived with Ssup (see Eq. 11 on Dmiz and Lunsup (see Eq. (1) on D.:
Ciill = (Cáá + c.upp + Au (Cumuup + ((nsup)
(13)
Update of Students and Teachers. Students are updated via gradient descent of Liall, while dual teachers are updated through the exponential moving average (EMA) of corresponding students:
0Crull
g n
0
- 000
8th + 38 + (1 -3)8%
(14)
The slowly progressing teachers can be regarded as the ensemble of students in recent training iterations, which enables stable progress in training.
4.3 Unbiased Superiority of Dual Mean-Teacher
For dense prediction tasks such as AVSL, employing pseudo-labels for model training can easily ac- cumulate eтrors and lead to sub-optimal results. In our DMT framework, The unbiased characteristics could be attributed to the following three factors: (i). Noise Filtering ensures that only stable samples are utilized to train. (ii). IPL. generates high-quality pseudo-labels. (iii). Pre-train dual teachers on bounding-boo annotated data with strong augmentation in Warm-Up Stage. The above conclusion
will be validated in subsequent ablation studies in Section 5.4
5

---

Table Comparison results on Flickr-SoundNet Models are trained on Flickr 10k and 144k. indi- cates our reproduced results, others are borrowed from original papers. Attention 1Ok-SSL is of 2k la- beled data supervision. We report the proposed DMT results from both stages as 2
[Dr] denotes the number of labeled data.
Flickr 10k
Flickr 144k
Methods
CloU
AUC
CloU
AUC
Attention 10k
14
43.60
44,90
66.00
55.80
CoarsetoFine
52.20
49.60
DMC 2
67.10
56.80
LVS 28
58.20
52.50
69.90
57.30
EZVSL 9
62.65
54.89
72.69
58.70
SLAVC [0]
66.80
56.30
73.84
58.98
SSPL 29
74.30
58.70
75.90
61.00
SSL-TIE' 30
75.50
58.80
81.50
61.10
Attention 10L-SLL 14.15
82.40
61.40
83.80
61.72
Ours ( [D; 256)
87.20
(84.40)
65.77
(59.60)
87.60 (84,40)
66.28 (59.60)
Ours [Di = 2k
87.80 (85.60) 60)
66.20 (63.18)
88 (85 60)
66.63 (63.18)
Ours [Dr = 4k
88.80 (86.20)
67.81 (65.56)
90.40 (86.20)
69.36 (65.56)
5
Experiments
With limited annotated data, DMT could significantly raise the performance of AVSL and address the severe issues, e.g., false positives and poor localization ability on small objects. Then, we direct our focus towards answering the following questions with ablation experiments 5.4
What is the individual contribution of each module to the performance gains?
How does annotation enhance localization performance?
Why can DMT outperform the existing semi-supervised AVSL method?
Is it necessary warm up dual teachers?
How to effectively mitigate confirmation bias in AVSL?
5.1 Experimental Settings
We conduct experiments on two large-scale audio-visual datasets: Flickr-SoundNet [4 15 and VGG Sound Source [46], where there are 5,000 and 5,158 bounding-box annotated samples, respectively. For labeled data, we randomly select 4,250 for training, 500 for validating, and keep the
Datasets.
same test sets with 250 samples as previous works 9 TO 2 30 Moreover, select subset of 10k and 144k unlabeled samples to train as before. Details can be found in Appendix B.I
Audio and Visual Backbones. For visual backbones, we followed prior work and used ResNet- 18 [47] pre-trained on ImageNet [48]. For audio backbones, select the pre-trained VGGish [49 and SoundNet (50) with semantic audio information. Further details can be found in Appendix B.2
Metrics. We report the Consensus Intersection over Union (CloU) and Area Under Curve (AUC), following previous settings 114 15). CloU represents the localization accuracy, samples with IoU above the threshold 5 = 0.5 are considered to be accurately located. Considering small objects, we introduce Mean Square Error (MSE), which measures the average pixel-wise difference between two maps without binarization, making it more suitable for evaluating dense prediction tasks on small objects. More details shown Appendix B.3
Implementation details. For audio clips, we pass 96 64 log-me mel spectrograms to VGGish, and the output is 512D vector, while the raw waveform of the original 3s audio clip is sent to SoundNet. For frames, we used an input image of size 256 x 256 x 3, with
224 512 as output. We choose RandAug (51) as strong augmentation, while random cropping, resizing, and random horizontal flip as weak augmentation. We set 6 as 0.6 and J as 0.7. More experiments of hyperparameters are shown in Appendix C.5

---

Table 2: Comparison results VGG-ss. Models trained VGG-Sound 10k and 144k.
VGG-Sound 10k
VGG-Sound 144k
Methods
CloU
AUC
CloU
AUC
Attention Ok 141
16.00
28.30
18.50
30.20
LVS (28
27.70
34,90
34.40
38.20
EZVSL 99
32.30
33.68
34.38
37.70
SLAVC 10
37.80
39.48
39.20
39.46
SSPL. 29
31.40
36,90
33.90
38.00
SSL-TIE 30
36.80
37.21
38.60
39.60
Attention SOk-SLL [0:151
38.60
38.26
39.20
38.52
Ours (D 256
41.20 (39.40)
40.68
(38.70)
43.60
(39.40)
41.88
(38.70)
Ours (Dr 2k
43.20 (42.60)
40.82 (40.75)
45.60 (42.60)
43.24 (40.75)
Ours (D 1k) 46.80 (43.80)
43.18 (41.63)
48.80 (43.80)
45.76 (41.63)
5.2 Comparison with the State-of-the-art Methods
Comprehensive experiments show that DMT achieves the sfate-of-the-ar performance among all existing methods on both datasets, and showcases several advantages.
Effective Utilization of Finite Annotations and Remarkable Performance. We tested DMT's localization performance with varying amounts of labeled data and found that it consistently outper- forms state-of-the-art methods when with 256, 2k, and 4k labeled data. Notably, even with just 256 labeled data, DMT achieved an accuracy of 87. 87.2% 87.6%, showing a significant improvement in CloU by around 10 absolute points compared to preceding models. Additionally, our model shows 3% absolute improvement in CloU compared maintains superior performance in complex and open environments. as demonstrated in Table 2 and Table 3c., indicating strong generalization capabilities. These results highlight DMT's ability to improve localization performance by utilizing more unlabeled data.
supervised-only model. Furthermore, DMT
Table Performance comparisons existing issues (small objects localization and false positives) and complex scenarios (open set). The results of small objects and open set are tested on the VGG-ss dataset, while false positives are reported on the Flickr dataset.
Methods
Small Testset Medium Testset
Methods
AP-
max-F11
Acc?
Methods
CloU
AUC1
MSEL
IoUft
MSE/
IoL't
LVS 28
17.90
19.60
LVS 28
26.30
34.70
LVS
0.515
0.021
DMC 2
25.56
41.80
$2.80
HZVSL 9
39.57
39.60
SLAVC 110
EZVSL 9
0.705
0.566
0.021
0.023
0.568
0.467
0.268
0.220
SLAVC 10
EZVSL 9
51.63
46.30
$4.60
59.10
83.60
66.40
SLAVC (10
38.92
41.17
0.160
0.025
0.174
0.335
Ours
53,56
62.80
91.60
Ours
43.12
42.81
(a) Small objects.
(b) False positives.
(c) Open set.
Significant Advancement in Small Subset Localization. We categorize objects based on their bounding box pixel area into small
322), medium (322 9622, large (962 1442) and huge (1442 ~ 2242). We tested different methods on small and medium objects in the VGG-Sound dataset, focusing on the challenges of detecting small objects and reducing false positives mentioned earlier. The results in Table 3a show that DMT significantly improves performance, especially in terms of MSE metric. Despite some errors in the IoU metric, DMT still outperforms previous methods. The results in Figure show that DMT accurately locates sounding objects with clear boundaries and precisely convergence to object contours, unlike previous methods that often have excessive or insufficient foreground regions, especially for small objects. These results demonstrate the effective and precise localization of small objects achieved by DMT. More experiments of different object sizes in Appendix C.7
Capability of Learning Rich Semantic Information. We present visualized predictions for testsets of varying sizes Figure [ (Left). It is evident that our approach achieves remarkable precision in localizing sounding sources. It accurately locates the position of sounding objects and precisely converges to their boundaries, while prior methods usually have excessive or insufficient foreground regions, particularly in the case of small objects.
7

---

Table 5: Main ablation study results. Models are trained on Flickr 144k and tested on eee testset. where S' and 'V' denote SoundNet and VGGish respectively.
Modules
Performance
# Teachers
Backbone
Filter
IPI.
EMA
CloUt
AUCT
MSF
(a).
X
X
80.20
53.57
0.458
One teacher
(b).
X
X
81.80
55.92
0.379
(c). S+V
X
X
82.20
55.16
0.382
(d). S+V
X
82.80
59.38
0,375
(e). S+V
X
X
83.60
62.83
0.359
Dual teachers
(f). S+V
X
84.80
65.58
0,259
(g) S+V
J
86.20
66.56
0.260
(h). S+V
X
86.60
66.35
0.274
(i). S+V
V
X
88.60
66.68
0.260
(). S+V
90.40
69.36
0.237
It is worth noting that our method can find out sounding objects overlooked in the manual annotations. For instance, in T 4-th row), the heatmap reveals the presence of a piano, which is omitted in the manual annotation process. Furthermore, we assessed the model's capability to identify false positives, signifying instances where sounding objects are occasionally not visually observable within the image (off-screen), as shown in 3b) This reflects the ability of Dual Mcan- Teacher to extract audio semantic information and effectively localize multiple sounding objects within scene, feat that eludes other methods. We attribute this capability the semantic alignment of audio-visual features achieved through the pre-trained VGGish and SoundNet backbone.
Capacity for Croos--Domain Generalization and Multi- Source Localization. We tested DMT's generalization across different domains and its ability to localize multiple objects. Models trained using VGG-ss 144k were directly evaluated on MUSIC-solo [37] MUSIC-duet (37], and
bbx
SLAVC
DMT
bbx
SLAVC
DMT
MUSIC-synthetic datasets 31 52 Figure@Jemonstratter
DMT's strong generalization performance in the music do- main, outperforming other method. As shown in Figure 3 the previous method struggles to accurately localize multi-
Figure 3: Performance on music-domain.
ple sounding objects, either missing them or including all sounding objects within a large foreground In contrast, DMT localizes each instrument accurately and separately. However, without category information for fine- mne-grained training, leads to sub-optimal performance in differentiating between multiple active and silent instruments. There is still significant room for improvement with multiple sounding objects and we plan to address this issue in future work.
5.3 Extensions of Dual Teacher
We replicated several existing methods and integrated them into our framework. Notably, the integration of the Dual
Table 4: Extension results of DMT with various
audio backbones, with 'R', 'V' and 'S' denoting ResNet, VGGish and SoundNet.
showcases its ability to sig- nificantly enhance the performance of other existing methods. In Table 4 one can observe noteworthy improvement in the CloU of EZVSL from 62.65% to 87.20%. and SLAVC rising from 66.80% 88.8 88.80% which further reinforces the efficacy of our frame- work and highlight its flexible extensibility.
Methods
Backbones
CloUt
AUC1
MSEJ
EZVSL we DMT
62.65
54.89
0.428
EZVSL w/ DMT
R+V
85.30
65.80
0.312
RZVSL w/ DMT
R+S
85.95
66.12
0.298
EZVSL DMT
V+S
87.20
67.74
0.256
SLAVC
SLAVC w/
DMT
R+V
86.10
66.24
0.288
SLAVC w/
DMT
R4S
86.30
66.58
0.283
SLAVC
DMT
V+S
88.80
68.69
0.247
5.4 Ablation Studies
What is the individual contribution of each module the performance gains? In this section, we progressively analyze the performance gain from each module in detail. We choose self- supervised approach as our baseline. Table presents the results on the Flickr 144k training set with 10% annotated samples.

---

(i). Initially, we apply it under the semi- supervised pseudo-labeling mechanism, using only one backbone, as shown in Table (Ta) and Table 5T.). The localization performance improves with annotated data supervision, but the gain is limited due to poor-quality pseudo-labels.
(ii) Next, we extend it to wwo-backhone architecture and sequentially introduce the filter, IPI., and EMA modules. The results demonstrate that all three modules contribute to performance improvement (3%, 1.8%, 1%). Notably, Filter shows the most significant impact on the model's improvement, for it effectively rejects noisy samples, ensuring the stability of the model.
(iii) Finally, can observe that optimal performance is achieved through the joint integration of the three modules by effectively suppressing confirmation bias.
How does annotation help localization? We aim to demonstrate that even with extremely limited labeled data, significant perfor- mance can still be achieved. To this end. we investigated the per- formance of our model from 0.5% to 10%. and report the results in Table b Our model consistently outperforms state-of-the-an ap- proaches across all ratios. Furthermore, with a constant amount of unlabeled data, as the proportion of labeled data increases, our model's performance continues to improve, highlighting the signif- icant impact labeled data.
Table 6: Performance on various labeled ratios % and multiple x Flickr 144k.
Labeled ratio %
CloU
AUC
0.5% (200/40k)
84.80
63.58
1% (400/40k)
86.20
65.16
2% (800/40k)
87.20
65.94
5% (2k/40k)
87.60
67.44
10% (4k/40k)
88.40
68.12
We investigated the impact of varying amounts of unlabeled data while keeping labeled data constant. Experimental results in Ta-
Multiple x
CloU
AUC
ble show that increasing the amount of unlabeled data improves localization performance, which seems contradictory the previ- ous conclusion about the proportion of labeled data, but actually, demonstrates that labeled data can effectively leverage the unla- beled data. Based on our analysis, labeled data not only provides annotation information but also effectively enhances the power of unlabeled data, resulting in significant performance improvements.
2.5% (4k/10k)
88.00
67.80
5X (4k/20k)
88.20
67.91
10X (4k/40k)
88.40
68.12
20X (4k/80k)
89.20
68.44
40X (4k/200k)
91.20
71.36
Why can DMT outperform the existing semi-supervised AVSL method? Both naive SSL and DMT have utilized labeled and unlabeled data. However, key distinction is that naive SSL employs unlabeled data only for contrastive loss, whereas DMT leverages pseudo-labels to incorporate unlabeled data into both contrastive loss and supervised loss, which amplifies the utilization of unlabeled data, thus enhancing generalization capability.
Data Utilization. We supplement the comparison experiments with fixed labeled data and an increase in unlabeled data from 10k to 200k, as shown in left part of 7 As the amount of unlabeled data increases, naive SSL exhibits only marginal improvement, whereas DMT shows more performance gains, indicating DMT can better use unlabeled data.
Generalization Ability. The right part of 7 Thighlights the limitations of naive SSL in the open set and in-the-wild datasets, suggesting that adding a supervised loss alone may lead to overfitting and weaken generalization. In contrast, DMT effectively leverages pseudo-label for improved generalization capability.
Table 7: Comparison of two SS-AVSI. methods. * denotes the results from the original paper. 'sim-avsl" denotes the simple self-supervised AVSL model we use. We report the CloU below.
2.5k/10k
2.5k/144k
2.5k/200k
open cross-datasets
attention 10k naive SSI.
84.00*/83.68
84.40*/84.08
84.24
19.60
62,20
attention10 DMT (ours)
88.00
89.52
90.40
42.64
87.26
sim-avsl + naive SSL
83.84
84.24
84.40
20.80
60.60
sim-avsl + DMT (ours) 88.24
89.76
91.12
43.10
89.80
Is it necessary to warm up dual teachers? We believe that the initialization of teachers and students is crucial, for the quality of pseudo-labels has a significant impact on model performance. To validate the effectiveness of this idea, we experimented to study the warm-up stage's 's impact on the model. We find that without the Warm-Up Stage, the model's 's improvement very slow, and the performance eventually deteriorates. This indicates that without good initialization, the model can

---

Quality Pseudo-labels
Effect of Different reudo-labels
Performance with Different Module
Filtered Samples Numbers
04
CloU of modd)
U modeel
0.7
number
266
73%
0.
04
64
Rauetla Samples
After Hiboing
fitoring
Teather &.Predttion
wel DMT Fiiter
DNT
DMT welo EMA
2000
4000
03
786
iteration
tetation
netastion
opoch
(a) Filter.
(b) IPL.
(c) Different modules. (d) Filtered number.
Figure 5: The effect of each component (Noise Filtering, IPL and EMA) in DMT to suppress
confirmation bias, together with the number of filtered samples for pseudo labeling depicted in (d).
accumulate errors, leading to confirmation bias issues. Therefore, we can conclude that Warm-Up is essential as it effectively suppresses confirmation bias in the early stages of training.
How to effectively mitigate confirmation bias in AVSL? In Section 4.3, we present the origins and mitigation strategies for confirmation bias in the localization task. In this section, will demonstrate this. Figure [5 depicts the quality of pseudo labels before and after applying the filter module. It is evident that the quality of pseudo labels can be significantly improved after filtering. This observation highlights the effectiveness of the filter in eliminating noisy samples, which tend to be
Effect of Warm-Up
100%
of model
80%
606
With Warm- Up
40%
Without Warm Up
2006
Warm-UU
false-posiin salse-positive instances. Figure (5) depicts the comparison be-
tween using the direct outputs of each teacher as pseudo labels and utilizing their intersection, known as IPI.. Where the pur- ple dashed line represents the initialization value. The results Figure 4: Effect of Warm-Up Stage.
1000
iteration
clearly indicate that employing the direct outputs alone leads to the accumulation of bias, causing deterioration in the quality of pseudo labels throughout the training process. Conversely, IPL consistently ensures the preservation of high-quality pseudo labels, thus mitigating the impact of bias.
Furthermore, Figure 5c visually presents the trend of model performance, revealing that the absence of any of these three modules results in decline in model performance. However, we can see that EMA only affects the final performance of the model, and without the Filter module. the model's performance will be significantly affected by noise. Without the IPL module, the model will experience a continuous decline in performance due to erroneous estimation of pseudo-labels. Therefore, we find that the Noise Filtering module and IPL modules play a significant role in addressing the confirmation bias problem. Moreover, Figure reflects that under the joint action of the three modules, DMT generates more accurate pseudo-labels and its performance continues to improve steadily.
6 Conclusion
In this paper, advance the naive SS-AVSL work and propose novel Semi-Supervised Audio- Visual Source Localization (SS-AVSL) framework, namely Dual Mean-Teacher (DMT), considering the importance of both limited annotated and abundant unlabeled data. From unified perspective, existing self-supervised (wcakly-supervisee) AVSL methods could be referred to as a single student structure, while DMT employs dual teacher-student pairs to filter out noisy samples via the agreement of two teachers and generate high-quality pseudo labels to avoid confirmation bias. DMT has greatly enhanced AVSL performance and addressed intractable issues like false positives and inaccurate localization of tiny objects. Moreover, DMT a learning paradigm and could be seamlessly incorporated into existing AVSL methods and consistently boost their performance.
We hope this work will bring more attention to SS-AVSL, provoke a reconsideration of pseudo- labeling, bias avoidance, and better utilization of the underlying unlabeled data, and thus stimulate more semi sumi-super learning research in this dense prediction task.
10

---

Acknowledgements
We would like thank the National Natural Science Foundation of China under Grant 61773374 and the Major Basic Research Projects of Natural Science Foundation of Shandong Province under Grant ZR20192D07. We also appreciate Shuailei Ma, Kecheng Zheng, and Ziyi Wang for their valuable and insightful discussions.
References
[1] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[2] Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He. Deep audio- learning: survey. International Journal of Automation and Computing, 18:351-376, 2021.
[3] Janani Ramaswamy. What makes the sound?: A dual-modality interacting network for audio- visual event localization. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4372-4376, 2020.
[4] Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. arXiv preprint arXiv:2008.0666 09622,, 2020.
[5] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pages 15516 15516-15525, June 2021.
[6] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visus separation of on-screen sounds. arXiv preprint arXiv:2011.01143 2020.
[7] Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In European Conference on Computer Vision, pages 368-385. Springer, 2022.
[8] Hanyu Xuan, Lei Luo, Zhenyu Zhang, Jian Yang, and Yan Yan. Discriminative cross-modality attention network for temporal inconsistent audio-visual event localization. IEEE Transactions on Image Processing, 30:7878-7888 -8888, 2021.
[9] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. arXiv preprint arXiv:2233.09324, 2022.
[10] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. In Advances in Neural Information Processing Systems, 2022.
[11] Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2023.
[12] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions pattern analysis and machine intelligence, 44(7):3614-3)) 3633, 2021.
[13] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. survey deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-20, 2022.
[14] Arda Senocak. Tae-Hyun Oh, Junsik Kim, Ming Hsuan Yang, and In So Kweon. Learning to localize sound source visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358 4366, 2018
[15] Arda Senocak, Tye-Hyun Oh, Junsik Kim, Ming-Hsuan Yang. and In So Kweon. Learning to localize sound sources visual scenes: Analysis and applications. TPAMI, 1655-166 2019.
11

---

[16] Dong- Hyun Lee et al. Pseudo-label: The simple and efficient semi-super learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, volume 3, page 896. Atlanta, 2013.
[17] Eric Arazo, Diego Ortego, Paul Albert, Noel E O' Connor, and Kevin McGuinness. Pscudo- labeling and confirmation bias in deep sumi-super learning. In 2020 International Joint Conference on Neural Networks (LJCNN), pages 1-8. IEEE, 2020.
[18] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
[19] Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning, Machine learning, 109(2)333--40,, 2020.
[20] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep semi-supervised learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[21] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Interna- lional Conference on Learning Representations, 2017.
[22] Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[23] David Berthelot. Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. T'lché-Bucc E. Fox, and Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[24] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol- ume 33, pages 596-608. Curran Associates, Inc., 2020.
[25] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. 2021.
[26] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
[27] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 2020.
[28] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16867-16876, 2021.
[29] Zengjie Song, Yuxi Wang, Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang. Self-supervised predictive learning: A negative-free method for sound source localization visual scenes. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1222-3231, 2022.
[30] Jinxiang Liu, Chen Ju, Weidi Xie, and Ya Zhang. Exploiting transformation invariance and equivariance for self-supervised sound localisation. In Proceedings of the 30th ACM Interna- tional Conference on Multinedia, pages 3742-3753, 2022.
12

---

[31] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33: 33:10077-10087, 2020.
[32] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 292-308. Springer, 2020.
[33] Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483-10492, 2022.
[34] Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, and Xiaowei Zhou. Visual sound localization in the wild by cross-modal interference erasing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1801-1809, 2022.
[35] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12934-12945, 2021.
[36] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross- agreement. 2020.
[37] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018.
[38] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1735-1744, 2019.
[39] Valentina Sanguineti, Pietro Morerio, Alessio Del Bue, and Vittorio Murino. Audio-visual localization by synthetic acoustic image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2523-2531,2021.
[40] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen- Yu Lee, and Tomas Pfis- ter. A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.0477 0475577777 2020.
[41] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end supervuue object detection with soft teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3060-3069, 2021.
[42] Gang Li, Xiang Li, Yujie Wang, Wu Yichao, Ding Liang, and Shanshan Zhang. Dtg-ssod: Dense teacher guidance for semi-supervised object detection. Advances in Neural Information Processing Systems, 35:8840-8855 2022.
[43] Qiang Zhou, Chaohui Yu, Zhibin Wang, Qi Qian, and Hao Li. Instant-teaching: An end-to-end semi-supervised object detection framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4081-4090, 2021.
[44] Yen- Yen-Cheng Liu, Chih-Yan Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, 7solt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint 21X.v22102.09480, 2021.
[45] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, arXiv preprint 1oX..0/610.0224 2016.
[46] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020.
13

---

[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[49] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Can architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icasxp), pages 131-135. IEEE, 2017.
[50] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in Neural Information Processing Systems, 2016.
[51] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[52] Di Hu, Yake Wei, Rui Qian, Weiyao Lin, Ruihua Song, and Ji-Rong Wen. Class-awar sounding objects localization via audiovisual correspondence. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(12))9844-9988 9859, 2021.
14

---

Appendix
Brief Introduction.
The appendix is structured into four main sections: Algorithm, Experimental Settings, Supplementary Experiments, and Further Analysis. The main contents are as follows:
Algorithm: Pseudo-codes and algorithm details.
Experimental Settings: More detailed description of the datasets, backbones, metrics formula, implementation details and baselines.
B
Supplementary Experiments: Results of cross-dataset evaluation, comparison of different predicted map, exploration of Warm-Up speed and its influence the final results, false-positive rejection capability of Noise Filtering, investigation of hyperparameters for Filter, IPL, and EMA, effect of data augmentation and quality analysis (visualization).
D Further Analysis: Theoretical elaboration on the challenges faced by existing contrastive learning methods, and explanation of why contrastive learning alone cannot achieve precise localization.
A Algorithm
To make it more clear, Dual Mean-Teacher is specifically depicted in Algorithm) |
Algorithm 1 Dual Mean-Teacher algorithm.
1: Input: D, = {aa e;),. Di {(v;, aa), 9.) (labeled data and unlabeled data.
2: while not reach the maximum iteration do
3: for (a,, 04) in D. do
4: while not reach the convergency of Warm- Up do
5: LWarm-Up (a (G,, P!) (Supervised learning on labeled data.)
6: end while
7: Get the pseudo-labels M M from dual teachers
8:
if IOU(ME .M.BB) 2 then
9:
IPL(a, v.) Mi..A..... MM (Compute Intersection of Pscudo-Labed (IPL).)
G IPL(a, u²) (Update the pseudo-label & of unlabeled data.) 11: Add (a,, v4) to new dataset D',
end if
13:
end for
Piix Di U D', (Mix the filtered unlabeled data and labeled data.)
Liill (Lg). + CB A, (Cassup + - (Students learning.) )
sup
6 <- -30m-1- (1 3)0%, (Students update teachers via EMA.)
17: end while
18:
Return: Dual teachers and students model parameters.
NOTING TIPS:
Train. Warm-Up Stage is essentially a supervised learning. The performance gains of subsequent Unbiased-Learming Stage over Warm-Up Stage is actually the performance gains of our semi- supervised framework over vanilla supervised training on the same labelled dataset D,, which proves the validity of the proposed Dual Mean-Teacher, shown in the main results in T Table 2
Inference. For the localization result of it audio-visual pair, we merge the outputs of the dual teachers to create predicted map as below. Comparison of different predicted maps are described
P. = (P..A + B.B.BB.. 2
(15)
15

---

B Experimental Settings
B.1 Datasets
We have conducted our training and evaluation of the Progressing Teacher on two large-scale audio- visual datasets: Ffickr-SoundNet and VGG-Sound, which consist of millions of unconstrained videos and 5, 000 and 5, 158 annotated samples, respectively. Each audio-visual pair is comprised of single image frame from each video clip and an audio segment centered around The annotations are provided in the form of bounding boxes. The relevant information is presented in the Table g
Table 8: Datasets overview.
All Labeled Data
Test Set
Labeled Split
small
medium
large
huge
total
small
medium
large
huge
total
train
val
test total
Flickr-SoundNet
3 254
687
4056 5000
0
9.
83
158
250 4250 500 250 5000
SGG-SeundSource
134
1796 1726
1502
5158
8
86
83 73 250 4250
500 250 5000
Furthermore, for the purpose of assessing the generalizability of our model, we have extended DMT to music domain (distribution), including: MUSIC-solo, MUSIC-duet, and MUSIC-Synthetic The MUSIC dataset (37) comprises 685 untrimmed videos, encompassing 536 solo performances and 149 duet renditions, spanning across 11 distinct categories of musical instruments. The MUSIC- Synthetic [31 [2] is a multifaceted assemblage wherein four disparate solo audio-visual pairs of divergent classifications are randomly mixed, retaining solely two out of the four audio segments. This deliberate curation aligns aptly with the evaluation of discemiingy sounding object localization.
B.2 Backbones: VGGish and SoundNet
For audio backbones, we employ pre-trained VGGish and SoundNet. VGGish is pre-trained on AudioSet as audio feature extractors. The raw 3s audio signal is resampled at 16kHz and further transformed into 96 64 log-mel spectrograms as the audio input. The output 128D vector. SoundNet takes the raw waveform of the 3s audio clip as input and produces a 1401D vector as output, which concatenates the 1000D object-level feature and the 401D scene-level feature, which are both obtained from different conv8 layer. Our main focus is to train the nonlinear audio feature transformation function, g(-), which is instantiated with two fully connected networks and a ReLU layer, to transform the network output feature into 512D representation.
B.3 Metrics: CloU, MSE, F1 Score, Precision
We consider set of audio-visual pairs as D = {(;, a.), Gi). where Gi is the ground-truth. We (P, (x, y) > 8} is the foreground region of predicted map, and Gi v) set P0) [x. y)
y) > 0} is the foreground region of ground truth.
CloU.
The IoU of predicted map and ground truth can be calculated by:
Ex.VP,, Ge(Z, v)
IoU,(8) Σx. GI v) [
(16)
In previous works, CloU quantifies the proportion of samples with IoU value exceeding a predeter- mined threshold, typically set at 0.5.
MSE. MSE measures the difference between two maps on a pixel-wise basis, making it more suit- able for evaluating dense prediction tasks than IoU, Other two metrics for small objects localization.
#
1 HW
MSE,
(P)(z. 2x)
(17)
16

---

Max-Fl and AP.
To compute true positives, false positives and false negatives, we closely follow SLAVC [0]. Then we can compute the precision and recall:
ITPI
ITP
Precision
Recall
(18)
TTP] + FPP
ITP + FN
Then compute Fl for all values of ó and report the Max-F1 score:
2 * Precision * Recall
Fl
max-Fl max(FI).
(19)
Precision Recall
Average Precision (AP) area under the precision-recall curve above. For a detailed calculation of max-Fl and AP, please refer to the SLAVC [T]
B.4 Implementation details
In addition to the experimental settings mentioned the main text, we used batch size of 128. Wa stage is trained for 6 epochs achieve convergence, while the Unbiased-Leamiin stage is trained for 20 epochs. The learning rate for the image is set to le-4, and the weight for the contrastive loss A, is optimizer is used for training. and the training is conducted on two GPUs. Our supplementary experiments conducted on the Flickr- 10k Flickr- 144k dataset, which contains 4k annotations. The trained models were evaluated on the Flickr-SoundNet testset.
1. An Exponential Moving Average (EMA) decay of 0.999 is applied. The Adam
B.5 Baselines
Attention 10k 14 15
(CVPR introduce dual-stream network and leverage an attention mechanism to capture the salient regions in supervieeri or self-supervised environments.
DMC 2 (CVPR2019) establish audio-visual clustering to associate sound centers with their corresponding visual sources.
CoarsetoFine (ECCV202)) 202C)) leveraged a two-stage framework capture cross- moosl feature alignment between sound and vision.
LVS 28 (CVPR2021): propose to mine hard negatives within an image-audio pair.
Table 9: Cross dataset performance. We train our model using the VGG-Sound 10k and 144k datasets and evaluate its performance on the Flickr-SoundNet dataset.
Flickr testset
Trainset
Methods
CloU
AUC
attention
52.20
50.20
LVS
61.80
53,60
EZVSL
65,46
54.57
SLAVC
74.00
57.74
VGG-Sound 10k
SSPL
76.30
59.10
SSL-TIE
77.04
60.36
Ours( D 256)
85.04 (80.08)
65.06 (60.14)
Ours( Di
2k)
87.3 (81.60)
67.38 (61.26)
Ours( Di
4k)
88.20 (82.88)
67.56 (62.06)
attention 10k
66.00
55,80
LVS
71.90
58,20
EZVSL
79.51
61.17
SLAVC
80.00
61.68
VGG-Sound 144k
SSPI.
76.70
60.50
SSL-TIE
79.50
61.20
Ours( D2 256)
87.04 (80.08)
64.72 (60.14)
Ours( (D 1 2k)
88.32 (81.60)
67.78 (61.26)
Ours( Di = 4k)
89.84 (82.88)
68.64 (62.06)
17

---

EZVSL 19 (ECCV2022) introduce a multi multi-instanee instance contrastive learning framework that utilizes Global Max Pooling (GMP) to focus only on the most aligned regions when matching audio and visual inputs.
SLAVC [10] (NeurIPS2022)) adopts momentum encoders and dropout to address overfitting and silence issues in single-source sound localization.
SSPL 29 (2VPR2022) propose a negative-free method to extend a self-supervised learning framework the audio-visual data domain for sound localization
SSL-TIE 30 (ACM-MM2vg2): introduce self-supervised framework with Siamese network with contrastive learning and geometrical consistency.
C Comprehensive Experimental Results
C.1 Cross-dataset Evaluation
To further validate the generalization ability of DMT, we conducted cross-dataset validation experi- ments. The results in Table D show that DMT still stays ahead, confirming the high generalization ability of our model.
C.2 Different Predicted Map
In this section, we compare the accuracy of different predicted maps for sound localization. We evaluate individual predicted maps and a fused map as the final localization map, as defined by Eq. [5 the Flickr144k dataset using dual teacher
Table 10: Results of different inference strategies.
CloU
AUC
Training is performed results, as shown in Table [O] We find that fused predicted map from dual teachers with different backbones achieves better localization performance than from individual maps, which can be attributed the fact that considering both localization results helps mitigate biases inherent in single model.
Student A Student Fused Students
86.20
66.16
86.80
66.84
88.60
68.56
Teacher A
87.20
67.57
Teacher 87.60
67.98
Fused Teachers
90.40
69.36
Additionally, assess the performance of teachers and students by comparing their fused predicted maps obtained during the same training session. The results, as shown in Table IO indicate that teachers outperform students, which aligns with expectations and further validates the effectiveness of our model.
C.3 Effect of Warm-Up Stage
Warm-Up Stage
100%
This section focuses on the analysis of convergence speed and the influence of the Warm-Up performance the final results.
CloU and AUC
009
Convergence Speed Initially, we investigate the convergence speed of the Warm-Up stage with varying amounts of labeled data, as depicted in Figure Notably, all supervised models exhibit rapid convergence within specific number of epochs. Furthermore, as the quantity of data increased, the convergence speed decreases while simultaneously achieving higher levels of performance.
Chnil
0%
epoch Figure Warm-Up.
Table 11: Effect of Warm- Up Performance.
Efffet of Warm-Up Performance. Subsequently, we investigate how the Warm-Up performance affects final results by experimenting with models that achieved different levels of convergence using the same amount of data. Training is performed on the Flickr 44k dataset using dual teacher results, as presented in Table II The results indicate that better performance of Warm-Up stage leads to better final model performance, which can be attributed to higher-quality pseudo-labels
Wann-Up
Final
CloU
AUC
CloU
AUC
84.32
64.52
51.20
48.62
87.28
67.18
71.60
56.08
89.04
68.26
and improved noise filtering, reducing confirmation bias. Conversely, the model exhibits the poorest performance in the absence of Warm-Up stage.
86.20
65.56
90.40
69.36
18

---

non-soundlnn frame
Teacher Teacher Prediction IoU of Dual Teachers 0.313
non-sounding frame
Teacher Prediction
Teacher Prediction
Prediction
loU of Dual Teachers 0.206
Figure 7:
Rejection Capability of Noise Filtering.
Overall, supervised audio-visual source localization demonstrates ease convergence without requiring excessive training resources. Moreover, our proposed semi-supervised model consistently outperforms the supervised model by approximately 3% terms of absolute performance, validating its effectiveness.
C.4 False-Positive Rejection Capability of Noise Filtering
After analyzing the filtered-out samples, we observed that the two independent teachers exhibit disagreement in localizing non-sounding objects. In such cases, the IoU falls significantly below the threshold, enabling the Dual Teachers to identify and reject non-sounding samples, which can be considered as false positives, illustrated in Figure 7 Additionally, different filter thresholds represents different levels of filtering strictness, as detailed in Section C.6
Furthermore, analyzed the visual results of some noisy samples, as depicted in Figure 10 One can observe that frames without distinguishable sound objects or sounds that cannot be accurately represented by bounding box (e.g., wind sounds) can be easily identified through the inconsistency between the predictions of the two teachers.
C,5 Hyper- Hlyper-parameters for Filter, IPL, and EMA
Effect of Pseudo-Labeling Threshold. The threshold is used to convert the predicted map into binary map, described (6) In this section, analyze the impact of different thresholds on pseudo-labels and the model. Training is conducted on the Flickr100 10k dataset. Figure 8 shows the results. small delta value (e.g. 0.5) creates large foreground area, introducing excessive noise and causing performance degradation as training progresses. On the other hand, A large value of 8 (e.g. 0.9) indicates small foreground area, causing the intersection between Dual Teachers to be minimal and resulting in samples being falsely rejected as noise, thus disturbing the model. Therefore, we choose 0.6 as the optimal threshold for our final selection.
Model Performance
Quality of Pseudo-Labels
Filtered Samples Numbers
of model
86%
number
86%
0.5
6 0.5
0.75
0.76
0.
0.5
3.6
iteration
iteration
epoch
(a) Model performance.
(b) Quality of pseudo-labels. (c) Number of filtered samples.
Figure 8: Results of various 8.
Efffet of Filtering Threshold. In Section 4.2, we employ confidence threshold, denoted T, to filter out noisy samples, which are more likely to be talse-positive instances. We evaluate the effect of different threshold values 7. As shown in Figure As the threshold value 7 increases from 0 to 0.9, the number of accepted samples decreases. However, setting very high threshold (e.g., = 0.9) leads to unsatisfactory results due to the limited number of accepted samples, reducing the available information from unlabeled data. Conversely, using a low threshold (e.g., 0.6) introduces confirmation bias from noisy samples, hindering favorable outcomes. Upon analysis, we discover that the performance shows little variation between threshold values of 7 0.7 and 0.8, indicating
19

---

balance between unlabeled information and bias within the 0.7-0.8 range. As result, we opt for 0.7 as the preferred threshold for our final selection.
Model Performance
Quality of Pseudo-Labels
Filtered Samples Numbers
model
15%
numba
80%
0.6
0.6
0.7
826
0.9
0.9
0.8
0.6
30%
iteration
iteration
epoch
(a) Model performance.
(h) Quality of pseudo-lahels.
(c) Number of filtered samples.
Figure 9: Results of various
We also examine the model performance with various exponential moving average (EMA) decay values, de- noted as 3, ranging from 0.9 to 0.999, and present the results of the teachers in Table 12 We observe that smaller EMA decay leads to faster update rate, lower CloU, and higher variance. Conversely, larger EMA decay value results in slower learning for the teachers. Therefore, we select an appropriate EMA decay value of 3 = 0.999 strike balance between the update rate and the stability of the learning process.
Effect of EMA Rates
Table 12:
Results on various
EMA B.
CloU
AUC
0.9
86.48
65.16
Flickr 10k
0.99
88.64
66.94
0.999
88.80
67.81
0.9
87.84
85.82
Flickr 144k
0.99
89.92
68.86
0.999
90.40
69.36
C.6 Effect of Data Augmentation
We evaluate the effect of RandAug [51 supervised model
4k labeled data, as shown in Table T3, Without data augmentation, the model exhibits significant over-fitting. With RandAug, this issue is mitigated, which indicates that RandAug a means of consistency regularization but also as method to enhance the model's generalization performance.
only
Table 13: Results of data augmentation (i.e., RandAug.).
Trainset
Testset
CloU
AUC
CloU
AUC
w/o Randnngment
88.20
67.82
84.80
60.44
w/ Rand-ugment
87.68
67.54
86.20
65.56
C.7 IPL on Different Object Size
assess the adaptability of IPL to various object sizes, and compare with existing methods, two teachers with DMT. Table [4 results highlight prior methods' diminishing performance with smaller objects, while DMT consistently excels across all size subsets. This enhancement attributed to Filtering and IPL synergy. Under the filtering mechanism, only highly similar pseudo-labels can contribute to model training. This keeps the intersection of pseudo-labels consistently aligned with object sizes. If pscudo-labels decrease significantly, IoU declines, excluding noisy samples from training. Moreover, the second-stage training, we use labeled data to prevent size bias and ensure unbiased treatment of objects of all sizes.
Table 14: Performance
various sizes of sounding objects.
SLAVC
teacherl
teacher2
DMT
Size
MSE 1
IoU
MSE )
IoU +
MSE 4
IoU 1
MSE 1
IoU f
small
0.705
2.10
0.213
2.58
0.183
2.26
0,205
2.65
medium
0.235
22.00
0.156
12.47
0.176
12.28
0.164
33.50
large
0.427
0.202
0.221
55.50
huge
0.358
61.64
0.212
66.84
0.217
66.26
0.215
67.70
20

---

C.8 How to avoid model collapse?
There are diversity and individuality between two teachers, as in Q2, which helps to prevent two teachers convergence to one model. The noisy filter module of DMT selects 'stable samples' via consensus and assigns high-quality pscudo-labels with IPL, such spirit has been validated by prior work that 'stable samples' could help avoid model collapse. Two teachers are first trained in Warm-Up stage for better initialization. Moreover, in stage-2, we also include supervised training labeled data and contrastive learning on unlabeled data, the two objectives would ensure the model possesses robust localization capabilities over the course of stage-2. 2. The results in Table 13 validate each component to avoid model collapse.
Table 15: Model collapse results. B denotes augmentation and backbone.
method DMT same A same 5 w/o annotation stage-2 same w/o annotation
CloU 90.4
87.2
85.44
81.6
7.2
C.9
Quality Analysis
We present the visual localization results of DMT in Figure 10 It effectively locates objects of different sizes, distinguishes them from the background by clear boundaries, and demonstrates some multi-object localization capability. Notably, DMT learns semantic information and can precisely localize specific sound-proden -rodducdd regions instead of the entire object. For example, in the third row of the Figure on on the right, it accurately locates the mouth of person rather than the entire person.
Ground Truth
LVS
EZVSL
SLAVC
Ours
Ground Truth
LVS
EZVSL
SLAVC
Ours
Figure 10: Visualizations of various methods.
D Further Analysis: Limitations in Existing AVSL and DMT
Based on the formula of contrastive loss, we can observe that the core idea of existing contrastive learning methods to match the visual frames and corresponding audio clips within the same video as a whole. The audio-visual pairs from the same video are considered positive pairs, while the frames and audio clips from different videos are considered negative pairs. The contrastive loss aims to maximize the similarity between positive samples and minimize the similarity between negative samples. The differences among existing self-supervised methods lie the selection the similarity function s(.) and the positive-negative sample pairs.
+ log (os(s((( Lunsup = -E(ayn))D, log ΣØ=1 exp (xp (g(;)) f(e,)) (T.)))TT)) /T.) Σ exp (s (f(())) g(a,)) /T)
(pp((g(aa).
21

---

D.1
Global and Local Information
In the given formula, different methods employ different match functions s(-) to compute the distance
or similarity between positive samples. For instance, Attention 10k [14 [5] uses the Euclidean
distance, LVS (28] utilizes the Frobenius inner product, and EZVSL applies Global Max Pooling:
Attention 10k: s(-) = Ilfau();)) g(ai)l2
LVS: s(-) Imppl (mip. sim (f(r))
EZVSL: s() max sim (f(v)),((a)))
sis (g ( (w:)) (
(siim (g (ai), fave (vi))
SLAVC:
s())
All of these functions capture the overall matching degree between audio and global visual repre- sentations. However, after the computation of s(-), the model loses the positional information of the two-dimensional visual representation. This positional information is crucial for fine-grained localization tasks.
D.2
Position-Aware Contrastive Loss
We refer to the methods that incorporate position information 'position-awaree. In the above formulas, we can observe that the distances or similarities between samples are calculated position-aware manner. For example, in the Attentionl0k [14 [5] method, the attention mechanism fau takes into account the positional information. Similarly, in LVS [28]. the foreground mask rijo distinguishes the background hard negatives, incorporating the positional context. EZVSL [9] uses the maximum value to capture the positional information, while SLAVC [T0] incorporates localization information. Taking LVS 22 an example, it specifically treats the background of the image as hard negatives, effectively leveraging the positional cues for discrimination and learning.
P; =
(maps sim(g(as).
mmppl
N; mminl
(1 min,
+ hwo iæi
[(.ssm(g(()))f(())))))
exp (P.)
Lunsup
log exp (P.) exp (N,)
where, mip is the mask of foreground, which strongly relies on the initialization of the model. According to the formula, both the positive (Pt) and negative (N,) samples in the training process influenced by the initial values of the foreground mask mip This implies that the model's localization results are heavily dependent the initialization.
D.3
Initialization
The different matching mechanisms, represented by the function s(-), rely the initialization of the entire visual model, specifically the pre-trained ResNet- 47 48), where the average of the pixel-wise features taken the initial result epoch O. This initialization result serves as the basis for the computation of position-aware components, such as the attention mechanism Global Max Pooling (GMP). Subsequently, during the model's training, these initial localization results are reinforced and refined. However, if the initial localization results are inaccurate (which is often the case), subsequent training may have difficulty detecting and correcting these inaccuracies. As result, the errors may accumulate over time without being effectively addressed, leading to degraded performance.
D.4 False Positives, False Negetives and Multi-Source
From the contrastive learning formula, it is apparent that contrastive learning assumes the presence of sound-producit objects in the visual input and enforces alignment between highly confident
22

---

visual regions and their corresponding audio features. However, pure contrastive learning, without the incorporation of additional modules, cannot directly reject non-sounding samples. Recently, some works have recognized this limitation and started to investigate the presence of hound-producing objects in images and tackle the task of multi-source sound localization. Examples of such works include DSOL [3T], IER 34 and AVGN I
Furthermore, due the absence of class labels during the selection of positive and negative samples, visual-audio pairs belonging to the same sound-prodd sound-producing object class but originating from different videos are still treated negative samples, resulting in false negatives issue. Several methods have emerged to address this problem, as highlighted in [35 36].
In addition, the commonly used matching mechanism, Global Max Pooling, is suitable only for single-source localization since it focuses solely on the region with the highest confidence, neglecting other potential nound-producd objects.
These three aforementioned challenges cannot be effectively resolved solely through simple models or algorithms without positional annotations. Therefore, they have become prominent rescarch areas that are currently receiving considerable attention.
D,5 Limitations DMT
DMT does not involve class information, so it struggles to localize among fine-grained objects due to poor discriminative ability. By incorporating category signals, models could better implement fine localization. Besides, DMT could not handle multi-object localization well. We will devise specialized components to address this issue.
23


Reviews:
[{"review_id": "1sJyyyDOx6", "reviewer": "Reviewer_GZse", "summary": "In this paper, the authors present a Semi-Supervised Learning framework, Dual Mean-Teacher (DMT), for Audio-Visual Source Localization (AVSL). DMT improves object localization in video frames paired with audio clips by employing two teacher-student structures to mitigate confirmation bias and generate high-quality pseudo-labels. The method notably improves AVSL performance, surpassing other methods on Flickr-SoundNet and VGG-Sound Source with only 3% of data being positionally annotated. The framework can also enhance existing AVSL methods.", "questions": "See Weaknesses.", "limitations": "The authors did not discuss method limitations in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper is easy to follow, and the method section is clear.\n\n+ The proposed semi-supervised method outperforms past unsupervised approaches and can be extended to boost different AVSL methods.", "weaknesses": "+ Semi-Supervised Audio-Visual Source Localization is not new. Back in 2018, Senocak et al. had already extended a semi-supervised approach to leverage labeled data to help with audio-visual source localization. I was surprised that the authors did not mention this in the paper. The experiments in the paper have larger benefited from the insight, experimental setting, and evaluation of the pioneering work.\n\n+ The effectiveness of the proposed Dual Mean-Teacher (DMT) is not fully validated. (1) The authors did not compare with existing semi-supervised AVSL methods (e.g., [1, 2]). (2) Existing semi-supervised learning approaches can be applied to solve AVSL. The authors should also add these naive A+B baselines into comparison. Without these comparisons, we cannot understand the significance of the performance improvements. For instance, in [2], the very weak Attention10k model with a naive semi-supervised learning trained on Flickr 10k can achieve 84% cIOU. The proposed method with a strong model and an advanced semi-supervised method under a similar setting achieves 87.80%. \n\n+ The technical contributions are limited. As stated in the paper, the key body of the used semi-supervised learning method is from existing work. The contributions of Warm-Up and Noise Filtering tricks are marginal. \n\n+ The demo only shows localization results on one example and did not compare with other approaches.\n\n+ How does the selection of teacher models affect AVSL performance? I think it is important to ensure that the two teachers produce accurate but different pseudo-labels. More experiments and analyses on teacher selection should be provided.\n\n [1] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. CVPR, 2018.\n\n [2] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to Localize Sound Sources\nin Visual Scenes: Analysis and Applications. TPAMI, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.\n\n> Q1: Semi-Supervised Learning (SSL) in AVSL is not new. Relationship to existing methods (Attention10k[1,2]).\n\n* Thank you for your suggestions. We have already cited and compared the unsupervised methods from [1,2] in the main paper, and will incorporate the semi-supervised results in [1,2] into the revised paper.\n* Pioneering works [1,2] introduced SSL to AVSL. However, the motivation, idea and contribution of DMT are novel and distinct from [1,2]. Please refer to **General Response**.\n* We summarize the core differences between DMT and Attention10k[1,2].\n  * Regarding SSL scheme, [1,2] naively employed $\\mathcal{L}_\\textrm{sup}$ on $\\mathcal{D}_l$, without utilizing pseudo-labels from $\\mathcal{D}_u$. In contrast, DMT introduced the pseudo-label mechanism to make full use of $\\mathcal{D}_u$ and addressed the confirmation bias issue.\n  * The dual-stram network in [1,2] serves as an unsupervised method, which is orthogonal to the proposed DMT framework, one could combine them to further boost performance.\n\n\n> Q2: The experiments in the paper have larger benefited from the insight, experimental setting, and evaluation of the pioneering work.\n\nWe adopted common setups for fair comparisons. We have also introduced our distinct settings. For instance, we introduced a new metric, MSE (Sec. 5.1, line 197), to validate small object localization. We also assess the quality of pseudo-labels quantitively (Fig. 4) and conducted experiments with various label ratios (Tab. 6).\n\n\n> Q3: The effectiveness of the proposed Dual Mean-Teacher is not fully validated. The authors should also add naive A+B baselines into comparison. \n\n* We appreciate your constructive advice. We devise the naive A+B experiments for Attention10k for more comprehensive comparisons. Here, 'naive SSL' means only employing $\\mathcal{L}_\\text{sup}$ on $\\mathcal{D}\\_l$ apart from $\\mathcal{L}\\_\\text{unsup}$ without any pseudo-labels. Results (CIoU) are shown below.\n\n|             method              | sup1k+unsup10k | suo2.5k+unsup10k | sup2.5k+unsup144k |\n| :-----------------------------: | :------------: | :--------------: | :---------------: |\n|          Attention10k           |      43.6      |       43.6       |       66.0        |\n|     Attention10k+naive SSL      |      82.4      |       84.0       |       84.4        |\n|              SLAVC              |      66.8      |       66.8       |       73.8        |\n|         SLAVC+naive SSL         |      80.4      |       82.2       |       83.8        |\n|          unsup (ours)           |      65.6      |       65.6       |       72.4        |\n|     unsup (ours)+naive SSL      |      81.2      |       82.8       |       83.6        |\n| unsup (ours)+naive pseudo label |      80.8      |       81.6       |       81.8        |\n|           DMT (ours)            |    **87.4**    |     **88.0**     |     **89.6**      |\n\n* **label ratio.** Results show that SSL could improve performance, and DMT surpasses naive SSL as $|\\mathcal{D}_u|$ increases. The proposed IPL enhances DMT with less confirmation bias and more utility of $\\mathcal{D}_u$.\n* **Generalization.** We compare naive SSL and DMT in terms of generalization ability to unseen objects, i.e., open set and cross-dataset results, as in **Rebuttal PDF**.\n* The results above highlight the validity of DMT and its advantages over naive SSL methods. We will incorporate these in the revised paper and integrate discussions concerning[1,2].\n\n\n> Q4: The technical contributions are limited. The contributions of Warm-Up and Noise Filtering tricks are marginal.\n\n* The central motivation lies in addressing the confirmation bias in SSL and achieving more efficient data utilization, which is under-explored in AVSL. DMT introduces a novel dual teacher-student architecture, harnessing the consensus of two teachers to rectify and generate high-quality pseudo-labels. DMT is a learning paradigm characterized by robust transferability and adaptability.\n* **Warm-Up** helps to avoid incorrect pseudo-labels from two teachers at the beginning and circumvent error accumulation[3]. We have empirically validated the necessity of warm-up in Sec. 5.4, Fig. 3, and Appendix C.3. **Filtering** is an online training method instead of simple data cleaning.\n* Ablations (Tab. 5) show that the absence of any module leads to significant performance degradation, e.g., without IPL, CIoU drops from 90.40 to 86.20, and without the filter, it decreases to 86.60.\n\n\n> Q5: The demo did not compare with other approaches.\n\nWe provide more comparison demos and the link is sent to AC as per the conference requirements.\n\n\n> Q6: How does the selection of teacher models affect AVSL performance?\n\n* The key to the success of dual teachers is that the two models are individual and will not converge to the same. Hence, we select distinct backbones and apply diverse augmentations to the two models, ensuring their independence.  In Table 4, we extend DMT to SLAVC and EZVSL, diverse backbones surpassed identical backbones. For instance, in EZVSL, (V+S) CIoU (87.20) outperforms (R+R) CIoU (82.80).\n* We have also added experiments with more combinations, please refer to the Table in **Response to Q3 of Reviewer dVkQ**.\n\n\n> Q7: The authors did not discuss method limitations in the paper.\n\nWe list the limitations in **General Response** and will add them to the revised paper.\n\n\n**References:**  \n[1]. Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. CVPR, 2018.  \n[2]. Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications. TPAMI, 2019.  \n[3]. Arazo E, Ortego D, Albert P, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning. IJCNN, 2020."}, {"review_id": "3uQtxTY56w", "reviewer": "Reviewer_13tz", "summary": "In this paper the authors address the AVSL task (Audio Video Source Localization), where given an image $v$ and an audio sample $a$, it is required to obtain a confidence map $P$ over $v$, which localizes in $v$ the object heard in $a$. Current approaches adopt self-supervised learning in the form of unsupervised contrastive learning: given a latent embedding $f(v_i)$ of the image divided in patches and a latent embedding of the audio clip $g(a_j)$, embeddings of naturally occurring pairs $(v_i, a_i)$ are brought together, with a loss $L_\\text{unsup}$ minimizing the distance between $g(a_j)$ and (a patch of) $f(v_i)$ and maximizing the distance between unrelated pairs. Such models have difficulty localizing small objects and suffer from blurry backgrounds. The authors propose adopting an unbiased semi-supervised learning scheme called Dual Mean Teacher, where a supervised loss $L_\\text{sup}$ is additionally employed during the learning phase, matching labeled boundary boxes with the predicted confidence maps $P$. The training proceeds in two stages. During stage one (Warm-Up Stage), they train two teacher AVSL backbones with $L_\\text{sup}$, using labeled data and strong augmentation for avoiding overfitting. During stage two (Unbiased-Learning Stage), they train two student networks following the Mean Teacher scheme (https://arxiv.org/abs/1703.01780) (using augmentation and updating the teachers with the EMA of the student weights). The student networks are trained both with $L_\\text{unsup}$ and $L_\\text{sup}$. For $L_\\text{sup}$, the labels are extracted from the labeled dataset or are given as pseudo-labels from the teachers, estimated on the unlabeled dataset. The author proposed the novel idea of using two teachers/students in order to mitigate confirmation bias: i. they perform noise filtering, namely when the two teachers output two pseudo-labels, whose IoU (Intersection over Union) is under a threshold $\\tau$, then the input sample is discarded as a false-positive. ii. By intersecting the output pseudo-labels (IPL), one can get a better estimate of the confidence map, and as such, the IPLs are the pseudo-labels fed to the students. Using this training scheme, the authors reach 90.4% CIoU on Flickr 144k and 47.6% CIoU on VGG-SS 144k, beating the competitor models by a large margin.\n", "questions": "- The authors mentioned that Instant-Teaching (https://arxiv.org/abs/2103.11402) uses two networks to correct pseudo-labels. What are the differences between the proposed dual Mean Teacher with the aforementioned method?\n - Given the proposed idea of a dual Mean teacher framework, why the authors preferred showcasing it on the task of AVSL instead of the more broad task of object localization? Is there a reason that such an approach is more naturally applicable to AVSL than to more basic object localization tasks? \n - Is there any relationship between the proposed model and ensemble methods? The idea is that ensemble methods create an ensemble with models that have been trained, and the ensemble is used at inference time. In this case, it is like dynamically ensembling the teacher models (and the students across the EMA updates).\n\n In the following, I add remarks and other misc questions for the improvement of the paper:\n\n- Line 48: Authors maybe are referring to Figure 2.\n - Line 94: Not being an expert in contrastive learning in AVSL, I had to find the correct reference to better gain an understanding of $L_\\text{unsup}$ (https://arxiv.org/pdf/2203.09324.pdf), maybe the authors could put a pointer for the readers in this section?\n - Line 96: Please define $\\mathcal{D}_u$ (it is defined later, but it should be here) and $\\tau$\n - Line 147: Using $\\tau$ again as on Line 96. Should use another letter to not make confusion.\n - Line 158: So if I understand correctly, consistency regularization is applied only on the filtered unlabeled input, and it is computed as the cross entropy between the IPL and the prediction of the students?\n - Line 159: I believe there is a typo because superscripts should be `s`.\n - Line 162: Authors could explain a little bit more in detail the attention module. Does it modulate the visual branch using the current audio-dense prediction?\n - Line 164: $D_u$, which could contain false positives, is still used when training the unsupervised branch?\n - Line 192: Put Appendix number.\n - Line 195: Please better explain what \"Considering previous methods often select the first 50% points of the entire image as foreground area of IoU, which is not suitable for small objects,\" means.\n - Line 199: Appendix in uppercase.\n - Line 204: What is $\\sigma$?\n - Line 207: No `:` at the end of the paragraph.\n - Line 210: Remove `when`.\n - Line 242: Why R+R? Shouldn't it be R+V or R+S?\n - Line 274: How do you get 200K if ds is 144k?\n - Line 290: Cannot understand this curve. Higher IoU means that the pseudo-label is better. In such case should filtering improve IoU?\n - Line 308: I don't understand what it means that we can filter when we don't have a filter.", "limitations": "The authors do not have limitations sections. The authors should discuss how to improve their method.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- To the best of my knowledge, the idea of using a pair of teachers/students in semi-supervised learning is novel, naturally realizing the idea of two different teacher models that can correct their estimates to provide better and unbiased pseudo-labels to student models. Such a novelty could be applied not only in the task of AVSL but to all cases in which a metric (IoU in this case) can compare different estimates, so as to remove false positives and where there exists a fusing operation (intersection in this case) that can blend to better estimates. \n\n- The authors successfully adapt semi-supervised learning to a task that has been classically dominated by unsupervised contrastive learning, efficiently integrating the usage of labeled data for predicting audio sources inside images. This is reflected in the strong quantitative results showcased in the experimental section, where consistently the method obtains the highest scores in all benchmarks. Using this training scheme, the authors reach 90.4% CIoU on Flickr 144k and 47.6% CIoU on VGG-SS 144k, beating the competitor models by a large margin.\n\n- Authors showcase qualitative samples that clearly better localize small objects with respect to other baselines.", "weaknesses": "-  It is true that the Semi-supervised second stage that integrates all the novel ideas proposed in a paper provides improvements in metrics (e.g., improving from 86.20% to 90.40% CIoU on Flickr 144k), but what I find odd is that already at stage-1, the obtained results beat all the baselines. This may indicate that \"supervised is all you need\" and that semi-supervised is not the key factor in improving the results (although methodologically the framework is very natural). The paper would definitely benefit from a more detailed scaling study, proving that indeed scaling to more unlabeled data, the semi-supervised mechanism is well exploited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.\n\n> Q1: \"Supervised is all you need\". Need more detailed scaling study with more unlabeled data.\n\n* Pure supervised-trained models suffer from severe over-fitting with poor generalization ability, while DMT could circumvent this issue by leveraging more unlabeled data with stable pseudo-labels. In addition to the results in our paper (Tab. 6 and Appendix C.1), we devised more detailed scaling studies as below (CIoU values).\n\n| Multiple  | train & test on VGG | train on VGG & test on Flickr | train & test on Flickr |\n| :------------------------------------------: | :-----------------: | :---------------------------: | :--------------------: |\n|                4k supervised                 |        43.80        |             82.88             |         86.20          |\n|                   (4k/10k)                   |        46.40        |             85.80             |         88.00          |\n|                   (4k/40k)                   |        47.60        |             87.60             |         88.40          |\n|                   (4k/80k)                   |        48.60        |             89.00             |         89.20          |\n|                  (4k/200k)                   |        49.80        |             91.40             |         91.20          |\n\nwhere values in multiple denote ($|\\mathcal{D}_l|/|\\mathcal{D}_u|$).\n\n* DMT shows **better generalization ability** than supervised training. From the table above, when the model trained on VGG and test on Flickr, pure supervision suffers from severe degragation compared with the one trained on the same dataset (86.20 drops to 82.88), while DMT maintains stronger generalization performance (86.20 drops to 85.80).\n* DMT more **effectively harnessing data**. Firstly, as the size of unlabeled data grows, DMT obtains consistent performance gains (10k to 200k, 85.80 to 91.40), which proves that DMT could benefit from more unlabeled data due to the stable pseudo-labels. \n\n\n> Q2: What are the differences between the proposed dual Mean Teacher with Instant-Teaching[1].\n\n* Firstly, instant-teaching implement SSL in a co-rectify manner, where one of the two teachers gives pseudo-labels to the other, while dual teachers in DMT collectively produce IPL for pseudo-labeling.\n* Secondly, there is no explicit instance-level noisy filtering mechanism in instant-teaching, while DMT filters noisy samples via consensus and only assign pseudo-labels to stable samples.\n\n\n> Q3: How about apply DMT to more broad task of object localization? Is there a reason that such an approach is more naturally applicable to AVSL?\n\n* In this paper, we focus on the novel AVSL where numerous challenges still persist, e.g, small objects localization, poor localization performance. The initial motivation of consensus filtering is to bypass false positives, which could also be use to filter other types of noisy samples.\n* The core spirit of DMT is valuable and generalizable, we will apply DMT to more general spatial (object detection) and temporal (event localization) localization tasks in the future.\n\n\n> Q4: Is there any relationship between the proposed model and ensemble methods?\n\n* Many existing ensemble learning techniques, such as bagging, random forest, primarily focus on ensemble predictions. These methods involve training multiple models independently and simply combining their predictions during testing.\n* The proposed dual teachers interactly with each other and collectively avoid confirmation bias by sample filtering and pseudo-labels, they influence each other instead of trained independently. \n* DMT is irrelevant to ensembling learning, there is interaction between dual teachers in training and inference, not simply ensembling the results or implement voting.\n* Also, refer to the Table in **response to Q1 of reviewer dVkQ**, DMT shows advantages over naive ensembling.\n\n\n> Q5: misc questions.\n\n* Ln. 48,94,96,147,159, 192 are typos, we will fix them.\n* Ln 158: Consistency regularization is applied on both the labeled and unlabeled data. In stage-2, we employ ground-truth label for labeled data and IPL for filtered unlabeled data. In both cases, students receive images of strong augmentation images while teachers with weak ones. Loss functions are cross entropy between the ground-truth labels (for $\\mathcal{D}_l$) or IPL (for $\\mathcal{D}_u^\\prime$) and the prediction of the students.\n* Ln 164: For $\\mathcal{L}_\\text{unsup}$, we also use filtered unlabeled data $\\mathcal{D}_u^\\prime$ to avoid false positives. Over the course of training, only $\\mathcal{D}_u^\\prime$ and $\\mathcal{D}_l$ are sent to the model. We will clarify it in the revised version.\n* Ln 195: In the literature of AVSL, among all the pixels, the pixel with the top 50% of the prediction confidence is defined as the foreground, then the foreground is employed to compute IoU with ground-truth bounding box.\n* Ln 204: It is a typo, here we mean $\\delta$.\n* Ln 242: We add 'R+V' and 'R+S' results, see below.\n\n|              |  R+V  |  R+S  |\n| :----------: | :---: | :---: |\n| EZVSL w/ DMT | 85.30 | 85.94 |\n| SLAVC w/ DMT | 86.10 | 86.30 |\n\n\n* Ln 290: Sorry, in Fig. 4 (a), there is a typo, the blue curve is 'After Filtering' while the red denotes 'Before Filtering'. Fig. 4 (a) proves that filtering module could impove the quality of pseudo-labels. We will fix the legend.\n* Ln 308: When there is no filter, the filtered numbers degenerate to the total numbers of the unlabeled data. We will refine it.\n\n\n> Q6: The authors should discuss limitations and how to improve their method.\n\nWe add the limitations, please refer to **General Response**.\n\n\n**References:**  \n[1]. Zhou Q, Yu C, Wang Z, et al. Instant-teaching: An end-to-end semi-supervised object detection framework. CVPR, 2021."}, {"review_id": "64fSaGzG7z", "reviewer": "Reviewer_dVkQ", "summary": "The authors propose a semi-supervised learning method for audio-visual source localization, which applies consensus-based confidence estimation to filter out noisy samples. The algorithm maintains two independent teacher models for the pseudo labeling of unlabeled data and for the consensus evaluation, where they use exponential mean average to update the two teacher models from two student models. Experiments using Flicker-SoundNet and VGG Sound source significantly improve performance from existing methods.", "questions": "In equation (1), how do you use the global max-pooing and average-pooling to pair f and g?", "limitations": "Do you use the same (augmented) input and network structure for the two student models?\nThe two models should have individuality and perform similarly to make the consensus meaningful. \nAre they optimal in this aspect, or is there room for improvement?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method significantly improves the audio-visual source localization performance.", "weaknesses": "While empirical improvement is significant, the technical novelty is limited. Obtaining performance improvement by model aggregation is a popular concept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.\n\n\n> Q1: While empirical improvement is significant, the technical novelty is limited. Obtaining performance improvement by model aggregation is a popular concept.\n\n* DMT is different from simple aggregation/ensembling. The primary objective of introducing dual teachers is to establish a filtering mechanism through consensus to mitigates confirmation bias. Dual teachers interact with each other and collectively filter select stable samples and assign high-quality pseudo-labels, as a result, the resulting models of both teachers are influenced by the other one.\n* To conclude, dual teachers in DMT interact with each other during training rather than simply ensembling the predictions.\n* Results below show that DMT outperforms naive ensembling of three models although DMT has only two models.\n\n|         method          | model 1 CIoU | model 2 CIoU | model 3 CIoU | ensembling inference CIoU |\n| :---------------------: | :----------: | :----------: | :----------: | :-----------------------: |\n| three models ensembling |     84.4     |     88.0     |     86.8     |           87.6            |\n|           DMT           |     89.2     |     88.8     |      -       |         **90.4**          |\n\n\n\n> Q2: In equation (1), how do you use the global max-pooing and average-pooling to pair f and g?\n\nI appreciate your highlighting of this concern. We will enhance the clarity in Ln 96-Ln 97. Here, 's' denotes the similarity criterion, i.e., cosine similarity. In the literature, visual inputs $v_i$  are typically transformed into features $f(v_i)$ via an encoder. The global features are obtained through global max-pooling or global average-pooling, where the pooling process is not explicitly shown in Eq.(1). We will clarify our writing accordingly.\n\n\n> Q3: Do you use the same (augmented) input and network structure for the two student models? The two models should have individuality and perform similarly to make the consensus meaningful. Are they optimal in this aspect, or is there room for improvement?\n\n* Two students employ different backbones with distinct prior knowledge and pre-training data. Besides, the data fed to the two students consists of different strong augmentation versions of the same instance. These two factors collectively ensure both diversity and individuality between the two models.\n* To ensure that both students approach the correct answer in stage-2 and preventing model collapse, we maintain the utilization of $\\mathcal{L}_\\text{sup}$ on $\\mathcal{D}_l$ and $\\mathcal{L}_\\text{unsup}$ on $\\mathcal{D}_u$. More importantly, the filtering mechanism efficiently sieves out noisy samples and mitigates confirmation bias. Here, we provide additional results on Flickr10k as follows.\n\n| Backbone |  R+R  |  S+S  |  V+V  |  R+S  |  R+V  |    V+S    |\n| :------: | :---: | :---: | :---: | :---: | :---: | :-------: |\n|   CIoU   | 84.0  | 84.8  | 85.2  | 88.6  | 89.2  | **90.4**  |\n|   AUC    | 63.06 | 65.58 | 65.72 | 66.42 | 68.20 | **69.36** |\n\n\n|      | same $\\mathcal{B}$, same $\\mathcal{A}$ | same $\\mathcal{B}$, diff $\\mathcal{A}$ | diff $\\mathcal{B}$, same $\\mathcal{A}$ | diff $\\mathcal{B}$, different $\\mathcal{A}$ |\n| :--: | :------------------------------------: | :------------------------------------: | :------------------------------------: | :-----------------------------------------: |\n| CIoU |                  84.8                  |                  85.4                  |                  87.2                  |                  **90.4**                   |\n| AUC  |                 65.38                  |                 65.96                  |                 66.28                  |                  **69.36**                  |\n\nwhere $\\mathcal{B}$ and $\\mathcal{A}$ denotes 'backbone' and 'augmentation' respectively. Results in the above two tables validate that distinct backbones and augmentations of two models make consensus meaningful and avoid confirmation bias."}, {"review_id": "En2J0nmEct", "reviewer": "Reviewer_MLW4", "summary": "This paper proposes a semi-supervised approach for audio-visual source localization on images based on an iterative student-teacher refinement scheme where the masks for the sounding objects are becoming gradually better by combining the estimated masks of two independent teacher models. The experimental results show that combining the estimates of the two teachers in such a way leads the student models to learn robust representations for the sounding objects under multiple datasets and yielding state-of-the-art results for the task of on-screen sound localization. The authors also provide experiments to show several ablation studies that show the importance of their design choices and the robustness of their proposed algorithm in different scenarios. \n", "questions": "How would the authors approach the problem of wanting to add more sound objects / actions that have not been seen during the pre-training stage of the first pseudo-mask estimators?\n", "limitations": "The authors discuss some limitations of their work and I think they should include the issues that I raised in the Weaknesses Section above and that they will not be able to address through experiments and/or rebuttal. \n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and the ideas presented are detailed enough to be followed by researchers in the field for reproducibility.\n- The experiments are extensive in multiple terms (ablation studies showing the importance of each design choice and each module, multiple datasets and metrics, including false positives and also analyzing the robustness of the algorithm).\n- The experiments show really strong results for the proposed semi-supervised approach for on-screen sound localization and show great potential for becoming a benchmark method in the field.\n", "weaknesses": "The paper has a great potential to be a valuable contribution to the field of audio-visual on-screen sound localization but it needs to address the following concerns / issues to uncover its full potential.\n\n- Using the word “unsupervised” is incorrect. The authors use this word to describe previous methods which all of them (to the best of my knowledge) still use pre-trained video/image encoders, pre-trained on the whole ImageNet and even sometimes some segmentation datasets. Thus, the authors should clearly distinguish all these works from what “unsupervised” truly means and properly define them as weakly or partially supervised on-screen sound localization approaches. This would not take away points from the novelty of this work . But in the current state, the manuscript’s multiple usages of the “unsupervised” word decrease its validity.\n- How do the authors enforce that the two teachers are going to be producing different sets of predictions? Is this simply the different teacher update protocols being used for the teachers? To the best of my understanding this training method could collapse if the teachers agree in terms of their predictions and converge both to a noisy pseudo-mask. This can be extremely cumbersome when one moves away from more curated datasets to more real world datasets with possible unseen sounding objects / actions.\n- Building upon my previous argument, using more realistic and large scale datasets would also show whether the proposed approach is robust when trained with videos that contain no on-screen objects which is very difficult to solve for related problems like on-screen sound separation [A, B]. I don’t want to extend the problem to cases where more than one sound appear on-screen / off-screen and thus some audio separation front-end might be needed before obtaining the audio features. But I would still expect some more experiments or discussion on how the authors could avoid potential problems like off-screen only videos with their method.\n\nI would be more than happy to increase my score if the authors address all my concerns and answer my questions.\n\n[A] Tzinis E, Wisdom S, Jansen A, Hershey S, Remez T, Ellis D, Hershey JR. Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds. In International Conference on Learning Representations 2021.\n\n[B] Tzinis, E., Wisdom, S., Remez, T. and Hershey, J.R., 2022, October. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In European Conference on Computer Vision (pp. 368-385). Cham: Springer Nature Switzerland.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.\n\n> Q1: Using the word “unsupervised” is incorrect.\n\nThank you for your advice. We will revise '**unsupervised**' to '**weakly-supervised**' due to the usage of pre-trained models and clarify the difference from truly 'unsupervised' learning. Besides, following your suggestion, we will emphasize that the task mainly concerns on-screen sound localization and supplement discussion regarding off-screen scenarios in our paper.\n\n\n> Q2: How to ensure two teachers will produce different sets of predictions? Is this simply the different teacher update protocols?\n\nDual teachers employ different backbones incorporated with distinct prior knowledge. Additionally, in stage-2, two students are fed the same image with two different strong data augmentations (pseudo-labels also require corresponding variations based on position augmentation), and in turn, they update two teachers via EMA. The aforementioned two points ensure the diversity between the two teachers.\n\n\n> Q3: How to avoid model collapse?\n\n* There are diversity and individuality between two teachers, as in Q2, which helps to prevent two teachers convergence to one model, which is validated in co-rectify paper [1].\n* The noisy filter module of DMT selects 'stable samples' via consensus and assigns high-quality pseudo-labels with IPL, such spirit has been validated by prior work [2] that 'stable samples' could help avoid model collapse.\n* Two teachers are first trained in WarmUp stage for better initialization. Moreover, in stage-2, we also include supervised training on labeled data and contrastive learning on unlabeled data, the two objectives would ensure the model possesses robust localization capabilities over the course of stage-2.\n* Results below validates each component for avoiding model collapse.\n\n| method | original DMT | same $\\mathcal{A}$ | same $\\mathcal{B}$ | w/o annotation in stage-2 | same $\\mathcal{A}$ & $\\mathcal{B}$ & w/o annotation |\n| :----: | :----------: | :----------------: | :----------------: | :-----------------------: | :-------------------------------------------------: |\n|  CIoU  |   **90.4**   |        87.2        |        85.4        |           81.6            |                         7.2                         |\n\nwhere $\\mathcal{A}$ and $\\mathcal{B}$ denotes augmentation and backbone.\n\n\n> Q4: More real world datasets with possible unseen sounding objects.\n\n* We have validated the effectiveness of DMT on real-world datasets, and in fact, VGG-ss is an in-the-wild dataset, encompassing various scenes and categories. As in Table 2, DMT achieves SOTA performance. However, there is indeed substantial room for improvement on real datasets, which is left for future work.\n* For the unseen objects localization issue, we have investigated open-set or cross-dataset generalization performance, as in Section 5.2, Table 3(c), we compare DMT with prior methods in terms of localization of unseen objects in the training-stage, DMT still achieves SOTA (CIoU 43.12 and AUC 42.81). Additionally, in Table 2 in Appendix, we have conducted cross-dataset experiments on unseen Flickr testset.\n* We also summarize the results here and add MUSIC reslts, as follows.\n\n|                   | openset |       | cross dataset |       | MUSIC |       |\n| :---------------: | :-----: | :---: | :-----------: | :---: | :---: | :---: |\n|      methods      |  CIoU   |  AUC  |     CIoU      |  AUC  | CIoU  |  AUC  |\n|    supervised     |  37.34  | 39.81 |     82.40     | 62.18 | 75.30 | 58.42 |\n| DMT w/o pre-train |  36.72  | 38.66 |     81.20     | 61.56 | 69.46 | 54.65 |\n| DMT w/ pre-train  |  43.12  | 42.81 |     89.84     | 68.64 | 82.12 | 61.86 |\n\n\n\n> Q5: Using more realistic and large scale datasets for off-screen scenarios.\n\n* DMT could leverage the consensus of two teachers to filter out off-screen samples, the results are shown in PDF file.\n* Off-screen is an important issue, we will add and cite papers [A,B] and take discussions in the revised version.\n\n\n> Q6: How to approach the problem with more sound objects / actions unseen during the pre-training stage of the first pseudo-mask estimators?\n\n* The pre-training gives backbones substantial prior knowledge, as it encounters numerous categories during the pretraining phase. The localization of unseen objects results degrade a lot if the backbone is trained from scratch, results are shown in **response to Q4**.\n* Models could leverage the similarity among objects to localize unseen but samples with similar semantics.\n\n\n> Q7: Include the raised issues unable to address through experiments.\n\nThanks for your advice. The limitations are shown in the **General Response** part and we will incorporate them in the future version.\n\n\n**References:**  \n[1]. Zhou Q, Yu C, Wang Z, et al. Instant-teaching: An end-to-end semi-supervised object detection framework. CVPR, 2021.  \n[2]. Ke Z, Wang D, Yan Q, et al. Dual student: Breaking the limits of the teacher in semi-supervised learning. ICCV, 2019.  "}, {"review_id": "CHHIxOrGoW", "reviewer": "Reviewer_1pQt", "summary": "The authors propose a two teacher-student architectures, pre-trained with limited labeled data in the semi-supervised learning framework to solve audio-visual source localization. They show that proposed approach can reach state-of-the-art performance in Flickr-SoundNet and VGG-SS. The authors also provide ablation studies and analysis in the characteristics of the dataset utilized.", "questions": "- In Section 4.2 the proposed intersection of pseudo-labels (IPL) already bias toward small regions. How do you make sure that there is no harm to large regions for proposed approach?\n- In Table 4, what is the datasets utilized for this comparisons?", "limitations": "This paper can benefit from applying to more datasets involving different audio domains and distributions of annotations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach to utilize semi-supervised learning techniques on audio-visual source localization is an interesting and original idea.\n- The authors provide thorough ablation studies on the choices of semi-supervised learning configurations, as well as backbones selections.\n- The analysis of data involved in this paper provides new insights and different perspective to the problem, including the analysis of small objects and false positives.", "weaknesses": "- Consider extend to more audio-visual source localization datasets such as MUSIC-synthetic, MUSIC-duet and AudioSet-instrument-multi described in the DSOL paper. This can help provide more insights on the proposed system in diverse of datasets including different audio domain such as music, and different distribution of sounding objects, number of bounding boxes in the image.\n- For the subsets mentioned in Section 5.2, it would be helpful to provide more definition on how small objects are defined, how false positives are measured, and the definition of open set.\n- Minor comments: in Section 5.2 line 211, consider explicitly refer those numbers to Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful advice and valuable questions, we will respond to your concerns point by point.\n\n> Q1: Consider extend to more audio-visual source localization datasets.\n* We have supplemented the evaluation on the music domain. Due to the absence of class information, we did not compare our approach with DSOL and its evaluation metrics.\n* Firstly, we directly evaluated the models trained with VGG-ss 144k on MUSIC-solo, MUSIC-duet MUSIC synthetic, and Audioset-multi, The setting and data splits follow DSOL [A]. We compare DMT with prior SLAVC, and SLAVC with naive semi-supervised learning (SSL), supervised learning, where naive SSL means models are trained on labeled and unlabeled data, with $\\mathcal{L}_\\text{sup}$ and  $\\mathcal{L}_\\text{unsup}$ respectively. Results (CIoU) are shown as follows.\n\n|     methods     | solo  | duet  | synthetic | audioset-multi |\n| :-------------: | :---: | :---: | :-------: | :------------: |\n|      SLAVC      | 73.28 | 6.00  |   3.86    |     26.74      |\n| SLAVC+naive SSL | 67.23 | 12.86 |   7.62    |     29.82      |\n| supervised DMT  | 75.30 | 9.66  |   4.17    |     28.16      |\n|       DMT       | 89.12 | 22.14 |   26.59   |     36.68      |\n\n\n* As shown above, DMT could generalize well in the music domain but with degraded results in multiple objects scenarios. From Figure 3 in rebuttal PDF, one could observe that DMT is capable of accurately localizing instruments in the image based, however, it lacks the ability to discern which instrument is producing sound and which one is silent based solely on the audio input.\n* We also finetune DMT on MUSIC-solo dataset, results in PDF show that DMT could gain partial improvements and is capable of finer-grained discrimination among certain instruments. However, due to the absence of class information, there is still significant room for improvement with multiple sounding objects. We plan to address this issue in future work.\n\n\n> Q2: Provide more definition of small objects, false positives, and open set.\n\nThank you for your advice. We will incorporate definitions for small objects, false positives, and open set. The revised part is as follows:\n* **small objects:** We categorize objects based on their bounding box pixel area. Small: $1\\sim 32^2$, medium: $32^2\\sim 96^2$, large: $96^2\\sim 144^2$, and huge: $144^2\\sim 224^2$.\n* **false positives:** Sounding objects are sometimes not visible in the image (off-screen). Such incongruous audio-visual pairs are often treated as positive samples in audio-visual contrastive learning, as they originate from the same video. Consequently, they become false positives.\n* **Open set:** It is important to investigae whrther model could generalize to unseen categories [B]. We randomly sample 110 categories (heard) subset from VGG-ss for training, and test our model on the other 110 categories (unheard) subset. The categories of the two subsets are disjoint.\n\n\n> Q3: Minor comments: in Section 5.2 line 211, consider explicitly refer to Table 1.\n\nThanks. We will refer Table 1 explicitly.\n\n\n> Q4: In Section 4.2 the proposed IPL already bias toward small regions. How to make sure there is no harm to large regions?\n\n* Firstly, IPL captures the intersection of pseudo-labels generated independently by two teachers. It employs the consensus of both models to rectify the pseudo-labels, yielding more accurate and high-quality labels, rather than progressively diminishing the intersection in order to bias towards small objects.\n* Secondly, both large and small objects have their appropriately sized pseudo-labels. Under the filtering mechanism, only highly similar pseudo-labels can contribute to model training. In this context, the intersection of pseudo-labels stabilizes consistently, maintaining proximity to the size of the respective objects. If the pseudo-label drastically reduces, the IoU decreases, preventing noisy samples from participating in model training.\n* Lastly, during the second-stage training, we continue to employ labeled data for supervised learning, which ensures the avoidance of bias towards small objects.\n* We have conducted experiments of various sizes. DMT shows significant improvements across all subset sizes. IPL in DMT is stable and non-detrimental. Results are shown below.\n\n|        | SLAVC |       | teacher1 |       | teacher2 |       | DMT  |           |\n| :----: | :---: | :---: | :------: | :---: | :------: | :---: | :--------: | :-------: |\n|        | MSE↓  | IoU↑  |   MSE↓   | IoU↑  |   MSE↓   | IoU↑  |    MSE↓    |   IoU↑    |\n| small  | 0.705 | 2.10  |  0.213   | 2.58  |  0.183   | 2.26  | **0.205**  | **2.65**  |\n| medium | 0.235 | 22.00 |  0.156   | 12.47 |  0.176   | 12.28 | **0.164**  | **33.50** |\n| large  | 0.427 | 48.11 |  0.202   | 55.32 |  0.221   | 54.68 | **0.212**  | **55.50** |\n|  huge  | 0.358 | 61.64 |  0.212   | 66.84 |  0.217   | 66.26 | **0.215**  | **67.70** |\n\n\n> Q5: What is the datasets utilized in Tab. 4?\n\nHere, we train and test on Flickr-10k.\n\n\n> Q6: This paper can benefit from applying to more datasets involving different audio domains and distributions of annotations.\n\nThank you for your advice. We have included experimental results across different domains, varying numbers of sound sources, and realistic datasets, along with appropriate citations to relevant papers. We have enhanced our analysis of the framework. For future work, we will focus on dataset results across various domains."}]

Author rebuttal:
We thank all reviewers for their dedication to our paper and insightful comments, and we believe these comments are significant for improving the overall quality of this paper.

We are pleased that the reviewers appreciate our paper from various aspects, including its strong performance [GZse] [13tz] [dVkQ], novelty [13tz], scalability [GZse], comprehensive experiments [1pQt] [13tz], clear and concise writing [GZse] [MLW4], and reproducibility [GZse] [MLW4]. These positive assessments truly motivate us.

The **motivation** of DMT is to solve the following problems:
*  The existing methods struggle to effectively utilize the abundance of unlabeled audio-visual pairs alongside the limited yet valuable annotated data.
* In semi-supervised learning, the issue of confirmation bias caused by low-quality pseudo-labels significantly undermines the training performance of the model.

Thus, the **core idea** of Dual Mean-Tacher (DMT) is to introduce a dual teacher-student structure, leveraging consensus from two teachers, incorporating a filtering mechanism, and utilizing the Intersection of Pseudo-Labels (IPL) module. This approach can rectifie and generate high-quality pseudo-labels online, effectively addressing the issue of confirmation bias.

We summarize our **contributions** as follows:
* We introduce a novel unbiased framework based on pseudo-label mechanism for semi-supervised AVSL, which could maximize the utilization of both labeled and unlabeled data, effectively address the challenge of limited annotated data and mitigates the issue of confirmation bias.
* Compared to existing approaches, DMT achieves much stronger localization performance, with better small object localization, handling off-screen issue and strong generalization capability. These strengths significantly elevate the performance of current weakly-supervised (previously referred to as 'unsupervised') methods.
* DMT can be summarized as a semi-supervised learning paradigm and could be combined with existing (weakly-supervised) AVSL methods to consistently boost their performance.
* The spirit of DMT could also be applied to other broad localization tasks.

The **limitations** of DMT are as follows:
* DMT does not involve class information, so it struggles to localize among fine-grained objects due to poor discriminative ability. By incorporating category signals, models could better implement fine localization. Besides, future work could consider employing huge pre-trained models, e.g. ViT pre-trained with DINO.
* DMT could not handle multi-objects localization well. We will devise specialized components to address this issue.

In this rebuttal, we have added **more supporting results** following the reviewers’ suggestions.

* comprehensive comparisons with supervised, existing SSL methods and naive A+B SSL baseline methods [GZse,13tz]
* scaling experiments with various unlabeled data [13tz]
* comprison ensemble learning baseline [dVkQ]
* model collapse analysis [MLW4]
* results on diverse domains including MUSIC and audioset [1pQt,MLW4]
* more visualization and ablation results on the proposed modules in DMT [MLW4,dVkQ]

Meta-review: The paper received mixed reviews initially. After rebuttal and discussion, the final consensus by all reviewers is to accept the work with the reviewers all being convinced and being positive (ranging from borderline accept to strong accept). The additional results and comparisons provided during rebuttal were quite useful in convincing the reviewers. It is recommended that the paper be accepted. It is strongly recommended to include the discussion and points provided during rebuttal in the final version of the paper.
Decision: Accept (poster)